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PREFACE 

The  Institute  for  Telecommunication  Sciences  (ITS)  in  Boulder,  CO,  (a 
laboratory  that  is  part  of  the  National  Telecommunications  and  Information 
Administration,  U.S.  Department  of  Commerce)  has  performed  the  study  reported 
here  with  funding  support  provided  by  the  U.S.  Army  Electronic  Proving  Ground 
(USAEPG),  Fort  Huachuca,  AZ.  This  support  was  provided  under  two  Military 
Interdepartmental  Purchase  Requests — TO  46-85  and  TO  47-85.  Administrative  and 
technical  guidance  to  this  study  was  provided  by  Mr.  John  Shaver  of  USAEPG. 

Tne  views,  opinions,  and  findings  contained  in  this  report  are  those  of 
the  author  only  and  should  not  be  construed  as  an  official  position,  policy,  or 
vision  of  the  'J.S.  Department  of  the  Army,  the  U.S.  Department  of  Commerce, 
•'  ir.v  other  agency  unless  so  designated  by  other  official  documentation. 

Tne  vuthor  is  particularly  indebted  to  Mr.  N.  B.  Seitz  and 
K.  ? .  '.infield  of  ^he  Institute.  Mr.  Seitz,  as  the  leader  of  a  group 
with  user-oriented  performance  standards  for  data  communications, 

;  velcpe  1  r.a.n y  of  the  original  concepts  applied  in  this  study.  Mr.  Linfield 
;  by  preparing  a  major  portion  of  Section  4.  Together,  these 

tgues  r.e’.ped  in  developing  the  initial  outline  for  the  study  and  provided 
,.r  ig^-.-r.r  and  technical  assistance  throughout  the  study  period. 
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INVESTIGATIONS  OF  TEST  METHODOLOGY  FOR  THE 
STRESS  LOADING  FACILITY 
R.  D.  Jennings* 

The  U.S.  Army  Electronic  Proving  Ground  (USAEPG)  is  planning  the 
development  of  a  new  test  facility  to  be  known  as  the  Stress  Loading 
Facility  (SLF).  This  facility  is  envisioned  as  an  integrated  and 
automated  test  capability  that  will  generate  a  dense  electromagnetic 
•  r.reat  test  environment  and  simultaneously  monitor  key  performance 
r v.' ".mc-ors  of  a  system  being  tested.  This  capability  is  expected  to 
:  -  ? erne  a  part  of  the  Electromagnetic  Environmental  Test  Facility 
EMETr  •  ,  both  physically  and  functionally.  However,  the  SLF  will  be 
:-sigre.i  to  provide  self-contained  operation  that  will  be  independent 
: th-'  HMETF,  if  required.  This  report  reviews  current  test 
ip  it . 1 : t  ies  that  are  relevant  to  the  SLF,  both  within  and  outside  of 
"FAFF  1 ,  and  develops  test  methodology  for  the  SLF.  The  test 
r.  -  do  logy  development  follows  a  structured  approach  in  the 
of  parameters  that  are  system  independent  and,  therefore, 

’.c'  used  to  describe  the  performance  of  various  systems  that  may 
r  f  2  using  the  SLF.  The  study  then  applies  the  structured 
;.r:  •  to  tr.e  development  of  performance  descriptions  for  two 
electronic  surveillance  systems  and  develops  the  associated 
.  •  r*’  crmance  measurement  methodology.  This  methodology  covers  test 
d,t-s  collect  ion,  data  reduction,  and  data  analysis. 

iut  mated  tests;  electromagnetic  compatibility;  electromagnetic 
v  .Ir.er.ibility;  development  tests;  EMC;  EMV;  field  tests; 

:  r.t  erferenoe  analysis;  laboratory  tests;  simulation;  SLF;  Stress 
Loaning  Facility;  system- independent  performance  parameters;  test 
:-j : gn ;  test  facility;  test  methodology 


1 .  INTRODUCTION 

r  Loading  Facility  (SLF)  under  development  by  the  U.S.  Army 
:  "ov ; r.g  Grour.i  'USAEPG)  is  envisioned  as  an  integrated  test  system 
r.reser.t  an  electromagnetic  threat  test  environment  to  systems  under 
w  . .  •  simultaneously  monitoring  the  responses  of  those  systems  by 
,•  performance  parameters.  The  SLF  concept  includes  the  use  of 

:  *  . .  *  -  gen'mcte  realisti  c  slices  of  time-ordered  electromagnetic 

1  r:  i - i c s ,  m.  control  all  aspects  of  real-time  test  and  SUT 
■  •  ,  •  ■/.::•  ;:*■  the  reduction  and  analysis  of  recorded  t°st  data. 

:  applicable  to  th<>  development  testing  (DT)  of  U.S.  Army 


1 nst i tut  ••  for  Telecommunication  Sciences ,  National 
«•"  «*  ior.  A  1m i hi  strut  ion,  U.S.  Dep  irt.ment  of  bommeree, 
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radio  frequency  (rf)  intercept,  direction  finding  (DF),  and  jamming  systems,  as 
well  as  radio  communication  and  radar  systems.  By  maintaining  positive  control 
over  the  test  process,  the  SLF  will  provide  the  tester  with  the  ability  to 
accurately  replicate  tests  or  portions  of  tests. 

1.1  SLF  Development  Program  and  ITS  Project  Goals 
As  noted  above,  the  Stress  Loading  Facility  will  provide  an  integrated 
testing  system  and  facility  to  achieve  a  dense  electromagnetic  threat  test 
environment  and  simultaneously  monitor  the  performance  of  the  system  under 
test.  Figure  1  presents  a  simplified  block  diagram  of  the  overall  SLF  concept. 
The  Communications  (COMM)  Threat,  Non-Communications  (Non-COMM)  Threat,  and 
Functional  Systems  Simulators  combined  with  the  Central  Computer,  Test  Control 
Station,  Test  Data  Monitoring  Subsystem,  and  appropriate  Interface  Unit  (to  the 
GMT'  comprise  the  SLF.  USAEPG’s  Electromagnetic  Environmental  Test  Facility 
: EMETF)  ( shown  with  dashed  lines  in  Figure  1)  is  not  part  of  the  Facility.  It 
will  be  important,  however,  to  integrate  the  SLF  and  the  EMETF  to  the  maximum 
extent  practical.  Capabilities  of  the  EMETF,  with  some  enhancement  of  current 
capabilities,  will  be  particularly  useful  in  preparing  deployment  data  and 
-echnical  characteristics  of  all  communications-electronics  (C-E)  equipment  for 
large,  complex  (both  static  and  dynamic)  test  scenarios. 

The  1 GMM  Threat  Simulator  (CTS)  will  be  used  to  replicate  the 
'■cmmunic.iticns  threat  environment  encountered  by  various  SIGINT/EW  systems. 
Tne  -.r.-CCMM  Threat  Simulator  (NCTS)  will  be  used  to  replicate  the  non¬ 
communications  threat  environment  expected  to  be  encountered  by  various 
."•1  CiNT.'EW  systems.  Each  of  the  simulator  subsystems  will  have  control  proces¬ 
sor  capabilities  to  operate  independently  of  the  Central  Computer  whenever  test 
conditions  or  Central  Computer  inoperability  dictate  a  need.  The  COMM  and  Non- 
Threat  Simulators  will  include  appropriate  modulators  and  rf  sources  with 
:c  mo  garner,  t  of  those  equipments.  The  Functional  Systems  Simulator  will  generate 
riate  "messages"  (functional  operations  is  a  more  generic  term)  for  the 
'  o.-;  or. -COMM  Threat  Simulators  to  use  in  managing  the  modulators  and  rf 

.-  s.  lot  Institute  for  Telecommunication  Sciences  (ITS)  has  accepted 
to  i-.il  ity  for  assisting  with  the  development  of  utilization  methodology 
r  .no.  ’ting  functional  performance  testing  of  complex  rf  systems  using  the 
To-  wore  undertaken  by  ITS  encompasses  three  tasKS  directed  to  develop- 
•  ••  f  i'.rs  met  node!  ogy :  (1)  review  existing  SLF-type  capabilities, 
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Figure  1.  Simplified  block  diagram  of  the  Stress  Loading  Facility  (SLF)  concept. 
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(2)  develop  measures  of  functional  performance  (MOFPs)  for  two  EWI  (electronic 
warfare  and  intelligence)  systems  selected  by  USAEPG,  and  (3)  develop  a 
framework  for  general  SLF  utilization  methodology. 

The  results  of  Tasks  1  and  2  are  presented  in  Sections  1  through  5  of  this 
report.  The  Task  3  results  include  application  of  the  functional  approach  to 
system-independent  testing  and  specific  application  of  the  methodology  to 
typical  electronic  surveillance  systems.  General  application  of  the  functional 
approach  includes  test  design,  data  collection,  data  reduction,  and  data 
analysis.  The  specific  application  to  typical  electronic  surveillance  systems 
includes  the  development  of  a  Detailed  Test  Plan  (DTP)  Outline  for  SLF  testing 
and  examination  of  interrelationships  between  bench  testing,  field  testing,  SLF 
testing,  and  computer  simulations.  These  topics  are  covered  in  Sections  6 
and  7  and  Appendix  C.  Conclusions  and  recommendations  as  a  result  of  this 
study  are  presented  in  Section  8. 

1.2  ITS  Approach 

A  structured  approach  that  establishes  uniform  methods  for  specifying, 
assessing,  and  comparing  the  performance  of  Army  C-E  systems  from  the  point  of 
view  of  functions  that  each  system  is  to  perform  as  defined  and  then  used  to 
develop  performance  parameters  and  measures  of  functional  performance.  This 
structured  performance  parameter  development  process,  illustrated  in  Figure  2, 
is  built  around  the  fact  that  a  C-E  system  is  designed  and  placed  in  service  to 
perform  a  number  of  explicit  functions  while  operating  in  some  expected 
environment.  The  system  (in  use)  will  perform  each  of  its  intended  functions 
with  only  one  of  three  possible  outcomes  being  realized  from  each  attempted 
use.  The  possible  outcomes  are  (1)  successful  (or  correct)  performance, 
'?)  incorrect  performance,  or  (3)  nonperformance.  It  must  be  recognized, 
however,  that  performance  in  the  aggregate  (over  time)  is  the  performance  of 
real  interest.  This  aggregate  performance  is  established  or  evaluated  on  the 
basis  of  a  sufficient  number  of  attempts  to  use  the  system  and  measure  the 
iiscrete  function  performance  under  known  and/or  controlled  environmental 
•seditions  so  that  performance  may  be  described  in  statistical  terms.  A 
comprehensive  understanding  of  system  performance  is  achieved  by  performing 
tests  using  various  factor  combinations  for  factors  (conditions)  that  influence 
system  performance,  as  discussed  in  Section  6. 
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In  considering  successful  performance,  one  is  concerned  with  the  time 
required  (or  speed)  to  perform  discrete  functions.  An  aggregation  of 
successful  performance  times,  then,  can  be  used  to  compute  statistics  such  as 
mean  time  for  successful  performance.  When  incorrect  performance  is  realized, 
one  is  concerned  with  the  frequency  of  incorrect  performance.  Given  a  suffic¬ 
ient  number  of  attempts  with  some  fraction  of  the  attempts  being  unsuccessful, 
on-1  car.  compute  the  fraction  of  all  attempts  that  were  unsuccessful.  The 
fruition  of  attempts  that  were  successful,  which  is  the  complement  of  the 
fraction  of  unsuccessful  attempts,  then,  can  be  computed  to  express  accuracy. 


If  nonperformance  is  realized,  one,  again,  is  concerned  with  the  frequency  of 
nonperformance.  The  complement  of  the  fraction  of  all  attempts  that  result  in 


formar.ee  is  reliability.  Measurements  from  a  sufficient  number  of 
s  can  be  used  to  compute  predictions  of  performance  outcomes,  to  some 


ir-.-l  level  of  statistical  -onfidence,  for  each  parameter. 


1.3  Report  Organization 


As  cited  in  the  preceding  subsection,  this  report  responds  to  three  tasks 
assist  wish  the  development  of  utilization  methodology  for  conducting 
fcr.ct  local  performance  tests  of  complex  rf  systems  using  the  SLF.  Section  1  is 
in  introduction  to  the  SLF  concept  and  the  methodology  development  support 


-■  v  1  C  ‘  fl  0  V 


Section  2  reviews  existing  test  capabilities  and  measures  of 


performance  used  by  non-USAEPG  organizations  as  well  as  those  within  USAEPG. 
Section  3  discusses  the  SLF  test  concepts  as  defined  currently  by  USAEPG. 
Section  H  outlines  and  discusses  the  structured  approach  that  ITS  has  followed 
in  defining  system  performance  parameters.  Section  5  describes  two  EWI  systems 
that  USAEPG  has  specified  for  this  study,  with  measures  of  functional  perform- 
i n o o  defined  for  each  system  in  accordance  with  the  structured  approach  set 
.*  o’Ch  in.  Section  for  defining  performance  parameters.  Section  6  describes 
•  nd  discusses  the  approach  to  performance  measurement  that  is  required  to 
sufficient  performance  data  to  allow  statistical  characterization  of 


to  some  desired  level  of  confidence.  Section  7  discusses  specific 


-m>-r.t  methods  for  testing  electronic  surveillance  systems  using  the  SLF 
;er  testing  capabilities  of  the  EMETF  and  examines  the  interrelationships 


SLF  testing,  bench  testing,  controlled  field  testing,  and  computer 
:  j I  a*  I  -n  that  pertain  to  complete  development  testing  of  Army  rf  intercept. 


indlng,  and  jamming  systems.  Section  8  presents  conclusions  and 
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recommendations  from  the  work  performed.  Finally,  Section  9  contains 
references  to  other  material  used  in  performing  this  study.  Acronyms,  abbre¬ 
viations,  and  unique  terms  used  in  this  report  are  defined  in  Appendix  A. 
Appendix  B  is  a  summary  of  the  structured  approach  applied  to  the  development 
of  performance  parameters  for  digital  communication  systems.  Appendix  C  is  an 
expansion  of  the  Detailed  Test  Plan  Outline  presented  in  Section  7. 


REVIEW  OF  EXISTING  TEST/MEASUREMENT  CAPABILITIES 


The  SLF  concept  incorporates  use  of  existing  test/measurement  capabilities 
o  the  maximum  extent  possible.  These  capabilities,  developed  by  organizations 


other  than  USAEPG  as  well  as  within  USAEPG,  are  reviewed  and  summarized  in 


■tions  2.1  and  2.2  respectively. 


2.1  Capabilities  Developed  Outside  of  USAEPG 
The  Naval  Research  Laboratory  (NRL)  developed  the  Tactical  Electronic 
Warfare  Environment  Simulator  (TEWES)  concept  in  1976  and  produced  the  first 
operational  system  in  1979;  this  capability  is  described  in  Section  2.1.1 
below.  In  conjunction  with  development  and  implementation  of  the  TEWES,  the 
NRL  also  has  developed  a  state-of-the-art  electronic  warfare  (EW)  facility 
known  as  the  Central  Target  Simulator;  this  capability  is  described  in 
Section  2.1.2  below.  RF  energy  coupling  (other  than  "antenna-to-antenna 
coupling"  techniques  that  are  suited  to  far-field  test  conditions)  using  near- 
Id  techniques  have  been  developed  and  are  used  for  testing  avionics  systems 
on  military  aircraft.  Such  techniques  have  a  number  of  limitations  that  will 
to-  espe-'ially  difficult  to  overcome  for  test  situations  such  as  those  for  which 
the  SLF  will  be  used.  Some  of  these  near-field,  rf  energy  coupling  techniques 


1  i soussed  in  Section  2.1.3. 


,ioel  Electronic  Warfare  Environment  Simulator 


r- :  above,  the  first,  operational,  NRL-developed  TEWES  system  was 
1  .  Since  that  time  a  variety  of  systems  have  been  developed  to 


'in-' i  •'  io  requirements  leading  to  the  Advanced  TEWES  that  became  opera¬ 
te  'f:5.  Ar.  Advanced  TEWES  (ATEWES)  currently  is  being  developed  by  NRL 


■J.’AKPG  to  become  part  of  the  overall  SLF  test  facility  (NRL,  1985). 

In*-  TEWES  is  i  gener-i I -purpose  simulation  system  for  evaluating  receiving 


P'-rition  1 1 1  •/,  tne  TEWES  allows  the  definition  of  realistic 
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Figure  3.  Functional  block  diagram  for  Tactical  Electronic  Warfare  Environment  Simulator  (TEWES). 


Table  1 .  General  Technical  Characteristics  of  the  TEWES 


SCENARIOS  (Time-ordered  event  lists)  .  4 +  Hrs 

PLATFORMS  (Participants)  . . .  256  Max. 

SIMULTANEOUS  SIGNALS  .  1023  Max. 


PULSE  DENSITY 


1,000,000  pulses/sec  avg. 
4,000,000  pulses/sec  peak 


SIGNAL  PARAMETERS 


Range/Resolution 


FREQUENCY  .  500  -  18,000/0.125  MHz 

AMPLITUDE  .  0  to  10  dBm( typical ) /I  dB 

ANGLE  OF  ARRIVAL  .  360/0.36  deg 

PULSE  INTERVAL  .  0.050  -  32,767/0.05  ysec 

SCAN  RATE  .  1  -  7,200/1  RPM 


PULSE  MODULATIONS  .  Stable  PRF,  PRI  Stagger,  Switching, 

Continuous/Discrete  PRI  Jitter, 
Continuous/Discrete  Periodic  Patterns, 

Synchronization 

RF  MODULATIONS  .  Stable  Pulsed,  Sequence,  Switching, 

Continuous/Discrete  Agility, 
Continuous/Discrete  Patterns, 
Multibeam,  Chirp,  CW 

SCAN  TYPES  .  Circular,  Sector,  Raster,  Conical, 

Helical,  Steady,  Omni,  Tracking 


2.1.2  Central  Target  Simulator 

The  Central  Target  Simulator,  also  developed  by  NRL,  is  a  state-of-the- 
art,  EW,  laboratory  facility  that  includes  a  three-axis  flight  simulator,  a 
centrally-located  computer  complex,  and  an  r f  environment  simulator.  Radiated 
rf  emissions  representing  multiple  moving  targets,  electronic  countermeasures 
(ECM),  and  environmental  phenomena  are  simulated  and  used  to  exercise  (stress) 
systems  under  test.  (TEWES  functionally  is  the  integration  of  these  items, 
except  the  three-axis  flight  simulator.) 

The  basic  CT3  facility,  additionally,  consists  of  a  shielded  anechoic 
chamber  with  one  wall  containing  a  large  spherical  matrix  array  of  antennas 
designed  t _  create  the  rf  environment  to  which  SUT's  are  subjected.  This 
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shielded  anechoic  chamber  creates  a  far-field,  free-space  propagation 
environment  for  the  radiated  rf  fields  (operating  in  the  8  to  18  GHz  frequency 
range).  The  size  (114  ft  x  127  ft  x  38  ft  high  or  3*».75m  x  38.71m  x  11.58m 
high)  and  spherical  geometry  of  the  chamber  enable  accurate  simulation  of 
tactical  environments.  As  observed  from  the  chamber  focal  point,  the  maximum 
field  is  78.75  deg  by  18.75  deg  in  relative  azimuth  and  elevation.  The  most 
recently  developed  facility  includes  only  225  antenna  elements  that  provide 
"coverage"  over  a  center  sector  that  is  8.75  deg  in  elevation  by  18.75  deg  in 
azimuth  plus  a  1.25  deg  strip  over  the  remaining  78.75  deg  in  azimuth.  Full 
field  coverage  would  require  800  additional  antenna  elements. 

The  USAEPG  SLF  will  require  a  facility  of  this  type,  but  with 
substantially  expanded  capabilities  to  accommodate  the  COMM  Threat  Simulator  as 
well  as  the  expanded  frequency  range  of  the  Non-COMM  Threat  Simulator.  Since 
the  expanded  frequency  coverage  (for  the  COMM  and  Non-COMM  Threat  Simulators) 
is  toward  lower  frequencies,  appropriate  facility  expansion  will  require  an 
unreasonably  larger  facility  unless  some  direct  or  radiated  near-field  coupling 
techniques  can  be  implemented  at  the  lower  test  frequencies. 

Radio  Frequency  Energy  Coupling  (Near-Field) 

SLF  testing  of  COMM  systems  will  require  rf  energy  coupling  using 
techniques  other  than  "antenna-to-antenna  coupling"  that  are  suited  to  far- 
field  test  conditions.  Near-field  techniques  have  been  developed  and  are  used 
for  testing  avionics  systems  on  military  aircraft.  Such  techniques  have  a 
number  of  limitations  that  will  be  especially  difficult  to  overcome  for  test 
situations  such  as  those  for  which  the  SLF  will  be  used. 

The  .near-field  coupling  test  applications  that  we  have  examined  are 
critically  dependent  on  alignment  of  each  antenna  with  respect  to  the  other  and 
the  separation  between  the  antennas.  These  physical  requirements  are  con- 

*  •-'tiled  through  the  development  and  use  of  elaborate  (and  expensive)  test  sets 
-hit  ensure  repeatable  test  conditions.  Other  physical  factors  that  will  be 
:  -  -‘.art  in  SLF  testing,  if  testing  is  attempted  using  antenna-to-antenna 

•  coupling  in  the  near  field,  include  transforming  the  SUT  performance 
;  1.-- 1  me  ter. a  for  near-field  test  conditions  to  expected  system  performance  under 
‘e’-f;-ld  'normal  operating)  conditions.  Factors  that  are  important  in  this 

include  signal  bandwidth,  phase  of  the  signal,  and  phase  relationships 
./stem  performance. 
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2.2  USAEPG  Capabilities 

The  U.S.  Army  Electronic  Proving  Ground  includes  a  number  of  capabilities 
for  testing  and  measuring  the  performance  of  communications-electronics 
equipment  and  systems.  These  capabilities  include  the  Communications  Data 
Measurement  Facility,  the  Scoring  Facility,  the  Spectrum  Signature  Facility, 

the  Radar  Weapon  Systems  Measurement  Facility,  and  the  Field  Facility.  A 

ut  or  automated  analysis  capability  complements  these  test/measurement 
••v.-.ri  itles  and  utilizes  performance  data  obtained  during  testing  in  these 

•  i  .  .  A  detailed  description  of  these  capabilities,  which  comprise  the 

•  ignetic  Environmental  Test  Facility  (EMETF),  is  given  in  a  USAEPG 

•  .  Brief  descriptions  of  these  capabilities  are  provided  in 

•  •  :  no  .  .  ’  through  2.2.3  which  follow. 

.  .  ■  Test ing  Capabilities 

:  .  -  i  an  purposes  only,  in  this  report,  the  Communications  Data 

Fa  :i  lity,  the  Scoring  Facility,  the  Spectrum  Signature  Facility, 

:  Wrap  on  Systems  Measurement  Facility  are  considered  collectively 

i  r.g"  capabilities,  since  the  Statement  of  Work  for  this 
■  :ev->lopmf:nt  study  asks  for  discussion  of  test  mode  interrelation- 
•  •  test  ing,  bench  testing,  field  testing,  and  computer 

.  •  .m  ;r.  i  "a  t  i  '  ns  Data  Measurement  Facility  is  used  to  measure  the 
performance  of  communications  links  subjected  to  selected  levels  of  an  inter¬ 
fering  si  .  The  rf  link  consists  of  a  transmitter  and  receiver  (for  the 

eve  ten  ;  rider  test  1  and  an  interfering  signal  source.  The  transmitter  of  the 
t  jnd*'-  test  is  placed  in  one  screen  room  and  coaxially  coupled,  through 
p •> i •, he  attenuators  and  a  mixer  into  the  receiver  of  the  system  under  test 
:  a-;c^nd  screen  room.  The  interfering  transmitter,  located  in  a 

•  ..  . . r m  t  a  1  so  is  coaxially  coupled,  through  appropriate  attenuators 

•  ,  into  the  receiver  of  the  system  under  test.  By  adjustment  of 

t r.e  degradation  to  performance  of  the  system  under  test  can  be 
‘".notion  of  the  desired  and  interfering  signal  levels  for 
'  that  vary  from  receiver  threshold  to  saturation.  This 
.  i.  with  appropriate  scoring  for  the  type  of  system  being  tested 

i. brat  ion  and  self-moni  toring  of  test  progress,  is  automated  to 
-d  it.e  tar---  tests  simultaneously,  as  illustrated  in  Figure  U . 

1  2 


he  parameters  normally  measured  Include  frequency,  power  levels,  spectrum 
'naraoter ist ics ,  transmitted  bandwidth,  occupied  bandwidth,  spurious  rf 
roducts,  and  modulation  parameters  of  the  system  under  test  and  the  interfer- 
ng  transmitter.  The  capability  is  known  as  the  Automatic  Data  Collection 

y stem  (ADCS). 

The  Scoring  Facility  includes  capabilities  to  score  both  analog  and 
igitul  communication  systems.  The  performance  of  analog  systems,  as  a 

ur.ct  i  or.  of  desired  and  interfering  signal  levels,  first  is  determined  by 
r  irtsr.it  ting  phonetically  balanced  words  over  the  system  under  test,  recording 
r.e  outputs,  an j  using  trained  listeners  to  determine  the  percentage  of  words 
h  it  are  correctly  understood  (scoring).  This  measure  of  performance  is  known 
w  ;r  1  intelligibility  or  articulation  score  (AS).  Subsequent  scoring  tests 
similar  systems  can  be  conducted  using  a  capability  known  as  the  Voice 
erf--rer.ee  Analysis  System  (VIA S).  In  this  system,  the  speech  frequency 
.'o:t”am  is  divided  into  1 A  equally  contributing  voice  power  bands.  The  noise 
•'  interference  in  each  of  these  bands  is  measured  relative  to  a  known  signal- 
:-rois--  ratio  to  determine  a  performance  score  known  as  the  articulation  index 
AT'1.  Tn-3  AI  scores  must  be  correlated  with  articulation  scores  (for  that 
ter  ,  but  the  VI AS  provides  a  convenient  and  automated  technique  for 
i mating  voice  intelligibility  for  an  analog  system. 

7 hi-  performance  of  digital  communication  systems  is  scored  using  a 
y.  ability  known  as  the  Digital  Scoring  System.  This  system  provides  automatic 
••il-tim-  measurements  of  the  bit  errors  in  the  digital  data  stream.  Measure- 
.-•r.ts  -■  -.a  include  bit  error  rates,  the  total  number  of  bits  transmitted,  and 
• r.  ;~hers  of  specific  typos  of  bit  errors.  A  typical  test  setup  that 
v,"  security  devices  and  error  correction  circuits  is  illustrated  in 

i  ••  jr--  ■  .  dote  that  digital  data  are  monitored  at  six  points  in  the  system  so 

•.  t *  da’  i  between  any  of  these  six  points  can  be  processed  in  analyzing 

•  ■-f  arm  of  the  system  or  portions  of  the  system. 

Tr.-.-  ''in  .'t.rur.  Sign  Facility  is  used  to  measure  detailed  technical 

.  ; r  :  : ‘  i  ■ t  i --n  of  transmitters,  receivers,  and  antennas.  The  measurement 
r  '.lit  inr’.-jd--  i  fixed  laboratory  with  screen  rooms  and  a  mobile  labora- 

-y  ‘  r  rs .  •-•••th  the  fixed  and  mobile  laboratories  are  equipped  to 

j.m  sign  itu  ”e  and  specialized  data  measurements  under  either 
g;  ■  - r . - f  i •  - 1  d  conditions.  Typical  transmitter  data  that  can  be 
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gure  5.  A  typical  test  setup  for  the  EMETF  Digital  Scoring  System  (a  "bench  test"  capability)  (L’SAEPG,  1987) 


--power  output 

--modulation  character ist ics 
--modulator  bandwidth 


— emission  spectrum  characteristics 

— intermodulation 

--carrier  frequency  stability. 


•al  receiver  data  that  can  be  measured  include: 


•sons  it ivity 
■overall  susceptibility 
•false  and  continuous 
wave  ( desensitization 
■  A 1  ■?  character i sties 


-audio  selectivity 
-selectivity 
-intermodulation 
-dynamic  range 
-discriminator 
bandwidth 


— spurious  response 
— adjacent  signal 
interference 
— oscillator  radiation. 


'.rar.smitter,  receiver,  and  antenna  gain  measurements  are  illustrated  in 


lata  obtained  in  the  Spectrum  Signature  Facility  are  used  as  input 


uter  simulations  in  addition  to  being  reported  as  empirical  data, 
span  System  Electromagnetic  Environment  Simulator  (WSEES), 
:n  Figure  7,  is  the  major  component  of  the  Radar  Weapon  Systems 


Facility.  A  second  capability  is  realized  by  combining  the  ADCS 


.vital  Scoring  System  with  the  WSEES  to  form  a  versatile,  electronic, 


,  testing  system.  Under  automated  or  manual  control,  the  WSEES 
generating  a  variety  of  rf  signals  that  include  pulse,  CW, 

■  .-mb  in  at  ion  of  pulse  and  doppler  CW,  pulse  burst  pattern,  chirp, 
:ng  to  represent  a  wide  variety  of  battlefield  C-E  systems  that 
frequency  range  of  2  to  18  GHz.  (Up  to  32  pulse  signals  can  be 
simulated.)  Control  capabilities  include  a  real-time  controller 
in  a  feedback  loop  with  the  system  under  test  so  as  to  produce  a 


nvironment  that  responds  to  the  electronic  counter-countermeasure 


r  :  lilies  of  the  system  under  test. 


>ting  Capability 


■  1 :  Facility  was  established  for  use  in  conducting  electromagnetic 
a!  t-estr  under  controlled  but  operationally  realistic  conditions. 


com:  r ;j  of  a  central  test  site  located  near  Gila  Bend, 


a!  outlying  sites  (situated  both  north  and  south  of  Arizona 


-'■  in vo  in  size  up  to  UO  acres  (see  Figure  3).  The  area  is 


:  shielded  from  the  urban  centers  of  Phoenix  and  Tucson  by 


test  instrumentation  facilitates  the  deployment  of 


testing  in  realistic  simulations  of  operational 
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test"  capability)  (USAEPG,  1987). 


Figure  7.  The  Weapon  System  Electromagnetic  Environment  Simulator  (WSE2S),  a  major  component  of  the  EMETF 
Radar  Weapon  Systems  Measurement  Facility  (a  "bench  test"  capability)  (USAEPG,  1987). 
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situations.  (In  cases  when  the  system  or  equipment  to  be  tested  cannot  be 
moved  from  its  installation  site,  the  mobile  instrumentation  vans  are  moved  to 
the  system/equipment  site  for  the  tests.)  The  Field  Facility  capabilities  are 
used  to  test  large-scale  deployments,  antenna  characteristics,  propagation 
factors,  over-sized  systems/equipment,  and  other  systems/equipment  that  cannot 
be  brought  into  the  laboratory  test  locations.  The  Field  Facility  also 
provides  realistic  conditions  for  acquiring  and  validating  data  in  support  of 
computer  simulation  analyses. 


2.2.3  Computer  Simulation  Capabilities 

Computer  simulation  capabilities  of  the  EMETF  consist  of  a  library  of 
computer  models  that  operate  on  a  dedicated  Cyber  172  computer  to  perform 
electromagnetic  compatibility  and  vulnerability  analyses  of  C-E  systems, 
equipment,  and  concepts  in  typical  field  tactical  environments.  The  library  of 
computer  models,  listed  and  described  briefly  below,  is  used  in  various 
combinations  to  perform  a  variety  of  electromagnetic  system  evaluations.  The 
library  includes: 


1.  the  Network  Traffic  Analysis  Model 

2.  t.ce  Performance  Analysis  of  Communications-Electronics  Systems 

(PACES)  Model 

3.  the  Intelligence  and  Electronic  Warfare  (IEW)  Model 

2.  t.ne  Spectrum  Integration  Model 

5.  the  Pseudoterra  in  Model 

b .  the  Frequency  Hopping  Model 

7.  the  Simulation  Model  for  Mobile  Subscriber  Equipment 

3.  the  Risk  Assessment  Model. 

The  Network  Traffic  Analysis  Model  is  a  time-  and  event-oriented  dynamic 
"peter  model  that  simulates  operation  of  individual  items  of  equipment  and 
ir  interaction  within  a  system  or  network  of  equipment  to  provide  network 
■iffi'-*  analyses  that  are  tailored  to  individual  problems.  The  performance 
al  rations  nay  focus  on  equipment,  links  and  nodes,  and  the  overall  system.  A 
ock  iiuvrvr,  of  the  Network  Traffic  Analysis  Model  is  shown  in  Figure  9.  The 
del  is  i  dynamic  and  event-oriented  simulation  that  is  driven  by  various 
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events  such  as  call  placements,  switchboard  connections,  and  equipment  failures 
that  are  processed  (as  part  of  the  simulation)  to  produce  new  events  called 
internal  events.  As  this  process  continues,  a  history  file  is  generated  from 
which  selected  data  are  extracted  to  support  specific  evaluation  requirements. 
The  simulation  uses  traffic  loading  and  flow  between  unit  types  and  offices  for 
each  tactical  deployment  as  defined  by  the  U.S.  Army  Signal  Center  and  School. 
Tactical  deployments  are  based  on  information  obtained  from  the  Communications 
I'.esearoh  and  Development  Command.  The  model  uses  a  static  tactical  snapshot  of 
;.;iprr.ent  and  equipment  locations  as  the  background  for  the  simulation  and  to 
reflect  the  dynamics  of  the  flow  between  items  of  equipment. 

The  vk:.Z$  Model  is  a  collection  of  programs  used  to  predict  the  EMC/EMV  of 
systems  and  equipments  operating  in  a  tactical  environment.  The  electro- 
\  i  ■  environment  is  represented  by  tactical  deployment  snapshots  that 
the  geographical  locations,  networks,  and  characteristics  of  the 
v  •  me  i.-.d  equipment  being  considered  according  to  situations  at  particular 
::.st  i r. a cf  time  in  a  force  model  sequence.  The  models  and  the  input  data  used 
?A Model,  derived  from  applied  theories  that  are  supported  where 
ih  le  by  empirical  data,  are  grouped  into  the  following  functional 

deployment  visibility  and  analysis 
preliminary  data  processing 
link  formulation  and  selection 
interference  identification 
probability  scoring 
svstem  effectiveness. 

d :  :--t.  of  the  PACES  Model  is  shown  in  Figure  10. 

1  r.yment  visibility  and  analysis  group  of  programs  provides  a 
:  •  p  -• en  t  » t  i cn  of  the  equipment  and  organizations  included  in  the 

•  •  :  it, an  ;  ver if  ios  "he  form  and  content  of  the  data.  These  programs 

-  v ;  t.  f'c-  data  necessary  to  convert  the  analysis  design  specifications  Into 

;•  *  a  i !  re  irenents  fo-  the  PACES  Model.  The  preliminary  data  processing 

•  f  rr  7r -  7 '  ■  >*-•>  ir.p  it  technical  data  against  the  deployment  require- 
:  ;  is  'irt'v  .rdit*  d  information  from  existing  files,  notes  missing 


information  that  is  not  in  the  file,  and  selects  the  data  required  as  input  in 
accordance  with  the  analysis  design  specifications. 

The  link  formulation  and  selection  group  of  programs  processes  the  input 
data  in  accordance  with  the  analysis  design  specifications.  A  statistical 
sampling  process  is  used  to  reduce  the  number  of  systems  and  equipments  being 
evaluated  to  a  practical  number.  This  sampling  process  is  weighted  so  as  to 
scl'»  ;t  the  systems  and  equipment  that  have  the  greatest  relative  importance  in 
i  - :  '"ip  1  i sh : r.v  the  assigned  tactical  mission. 

T-.e  interference  identification  group  of  programs  calculates  the 
s‘  .=  ‘  :s-  log  ->f  the  desired  and  interfering  signal  power  levels  at  the  input 
term s  1  •  f  the  link  receivers  being  evaluated.  Transmitters  that  are 
:•  •*  ••rt  i  al  mtorferers  to  each  system  being  evaluated  are  determined. 

trot  ability  scoring  group  of  programs  determines  the  probability  of 
.aeration  for  analog  links  and  the  probability  of  bit  error  for 
.  •; r.K s  selected  for  evaluation,  based  on  desired  signal  power  level, 
i  r: transmitters’  power  levels,  receivers’  characteristics,  and  duty 

:  interfering  transmi tters.  The  system  effectiveness  group  of 

:  r  ,;r  its  aggregates  link  performance  probabilities  in  accordance  with  the 
.  • ;  r.  v ,  s  :-'Sign  specifications.  The  system  effectiveness  score  then  is 

-.a  sod  on  the  relative  weight  assigned  to  each  link  for  success  of 

\  1  .  1  V.  :''V  1  b  3  1 0  Pi  • 

:••••  intelligence  and  electronic  Warfare  Model  is  an  analytical  tool  (which 
*.:>  r  either  the  CYBER  1 80  or  a  VAX  11/780  computer)  designed  to  evaluate 
;  ;  v.’ it  itivel;  the  effects  of  electronic  warfare  on  combat  operations.  The 

:•  !  ;  ■  a  :yr.ar.in,  event-driven  simulation  of  the  operation  of  a  set  of  sensor 

:  at ig  t  i.nst  an  opposing  force's  electromagnetic  environment.  A  block 
:L  igram  ■!  T.o  IFW  Model  is  shown  in  Figure  11.  Operation  of  the  sensor 
•  o  n  '■:!••  led  as  sets  of  discrete  events,  either  external  or  internal, 
‘h  t*  m  r ••inv-5  in  time  when  a  significant  change  occurs.  External  events  are 
f  •  m  •••itsi  Je  the  simulation  and  result  in  a  baseline  loading  for  the 
"  .  1  -  *■•:'' ra  1 1  events  are  generated  within  the  simulation  and  represent 

o.  s.v-0  •  f  r. sor  operation  and  target  operations.  There  are 
••  ..  modules  in  the  model — the  emitter  environment  module,  which 

•.  i  1 1  m  ac  *■>>'”  i  st  i^s  and  network  organization  of  the  emitters  in 

•  ■  o'  :  * h  •-  s-n.-  module,  which  generates  the  tactical  message  that 
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each  sensor  reports  during  the  simulation;  and 
provides  a  dynamic  capability  to  the  simulation. 

The  Spectrum  Integration  Model  is  used  to  calculate  threshold 
s  i  gr.a  i-to-i  nter  f  erence  (S/I)  values  when  empirical  data  are  unavailable 
"B"  value  scoring  data).  Input  data  for  the  model  include  the  interference 
spectrum,  the  receiver  selectivity  curve,  and  the  cochannel  threshold  S/I  value 
for  each  system  of  interest  and  the  frequency  differences  between  the  intei — 
: -hi  the  receiver  tuned  frequencies  for  the  desired  systems.  Details  of 
to  '  ulation  process  are  given  in  the  USAEPG  (1937)  report. 

The  oilculaticn  of  propagation  path  losses  between  transmitters  (desired 
interfering'  and  receivers  is  a  central  requirement  in  estimating  EMC/EMV. 
T  a  o  •me  P •'•_>, idoterra in  Model  is  given  to  the  model  that  performs  this  vital 
The  model  used  for  the  propagation  loss  calculations  is  the  Longley- 
;  ”  :-i  revised  in  1972)  with  modifications  made  by  the  EMETF.  The 

Model  is  one  of  the  better  models  for  predicting  long-term 
■.  ,  moo.  in  radio  transmission  loss  at  VHF  and  higher  frequencies  over 

•.■•■•’'iin  that  is  characterized  by  the  use  of  statistical  descriptors 
f  t irregularity,  surface  refractivity,  etc.  The  model  is  based  on 
•••■  1 1 no  L  ished  propagation  theory  and  has  been  verified  using  a  large  number 
f  ”  :  a  *  iticr.  measurements.  The  "heart"  of  the  model  is  the  calculation  of 
••■11  in  v  ilu  -s  of  reference  attenuation  relative  to  free-space  loss  as  a 
>.  ,f  distance  and  the  type  of  radio  path,  i.e. ,  line-of-sight,  diffrac- 
i  n,  r  t •’  c cspher  1  :  scatter.  The  median  basic  transmission  loss,  a  function 
::  •  -.C"-,  is  combined  with  other  parametric  values  that  account  for  the 
.••r’ i  -r:  lit •*  in  transmission  loss  due  to  long-term  fading  (time  availability), 

; Td-no-riid  variations  ( location  variability),  and  estimating  (or  prediction) 
i  "acu  variability  is  assumed  to  be  approximately  normally  distrib- 

:  » :  th  ••no  mean.  The  standard  deviation  for  each  variability  (denoted  as 
•- ,  ■;  ,  r.:  r.as  te»n  determined  empirically  from  measured  data. 

The  !•■'•••  ;e.ncv  Hopping  ‘-'odel  contains  logic  to  represent,  realistically, 

:  h  :.pr  in?  systems  in  a  deployment.  The  three  conditions 

i  no  1  d  -r-n  ; -n  of  frequency  hopping  systems  is  important  are: 

t  ■  i  .f  interference  from  frequency-hopping  systems  to 


>  r  : 


■nm  nonhopping  systems  to 


the  effect  of  interference  from  frequency-hopping  systems  to 
other  frequency-hopping  systems. 


effect  of  interference  from  nonhopping  systems  to  other  nonhopping  systems 
*he  situation  normally  considered  by  the  PACES  Model.  The  weighted  influ- 

■  j  t.  system  performance  scores  when  frequency-hopping  systems  are  involved 

f-rers  and/or  receivers  are  determined  by  the  Frequency-Hopping  Model, 
v-  M-.ulation  Model  for  Mobile  Subscriber  Equipment  is  an  event-sequence 
simulates  mobile  subscriber  equipment  system  functions  that 
basis  from  which  to  predict  the  EMC/EMV  of  mobile  C-E  equipments 
.  ”  •  the  predictions  calculated  by  the  EIEM  for  nonmob ile  systems  and 
.  .  The  model  includes  modified  algorithms  for  propagation  loss 

.  i rd  equipment  performance  criteria  that  are  tailored  to  mobile 

.  •  -c.ipment  operations  and  functions.  The  analysis  output  data  are 

levels  cf  detail-- ind iv idual  communications  link  scores 
-  arid  comp  at  i  Pi  lity/vulnerabi  1  i  ty  scores)  and  composite,  system 

-as  scores  for  sets  of  links  under  specific  interference  conditions. 
I:.-  i---:  Assessment  Model  is  used  to  evaluate  the  effects  of  unintended 

■  •  . -  : 1 i t i  -s  and  emissions  on  the  electromagnetic  compatibility  of  systems 
;  . : .  m-.-r.t  operating  in  their  intended  electromagnetic  environments. 

•  -.a  and  equipments  that  are  purchased  by  the  U.S.  Department  of  Defense 

■  ;.:re:i  to  operate  in  accordance  with  MIL-STD-1J61  /  462  /  46  3  with  regard  to 
■  •  m  acr-.-'t  ic  characteristics.  Susceptibilities  and  emissions  that  fail  to 

•  u  ireme.nts  of  these  standards  may  or  may  not  affect  the  electromag- 

!  -mrat ibility  of  the  system/equipment  when  deployed  in  the  intended 

■  .  n  !  1  environment.  This  model  is  used  to  develop  a  basis  for  a  decision 

t  system/equ ipment  or  to  field  the  system/equipment  without  change 
a  Is  some  s  isceptibilities  and/or  unintended  emissions. 

.oi-icjs  computer  simulation  capabilities  are  operational  on  a 
:*  •  •  •  d  '/Mod-M  3 30  computer  that  has  been  installed  in  a  classified 

•  .  ■  -.p;  r  ’V"  !  for  handling  and  processing  classified  information. 

■V:  •  'ientr  il  memory  'HAM)  of  51^,000  60-bit  words  ( 5 , 1  ^ 1  ,000 
•  •  .--isoo  cycle  time,  which  equates  to  a  little  more  than 
’  -  . : .  um<  r  i  a  characters  of  fetch-and-retr  ieve  cycles  from  RAM  per 

.  .v-r  c*«na*.i  1  i ty  is  illustrated  in  Figure  12. 
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■.  STRESS  LOADING  FACILITY  ( SLF )  TEST  CONCEPT 
lescription  of  the  SLF  is  given  in  Section  1  with  a  simplified 
of  the  SLF  concept  shown  in  Figure  1.  Additional  description  and 
the  SLF  test  concept  is  given  in  this  section,  but  reference 
;rv  limited.  The  purpose  of  this  section  is  to  show  (our  under- 
vow  existing  and  newly  developed  testing  capabilities  will  be 
d  automated  further)  to  achieve  the  SLF  testing  capability, 
ted  earlier,  the  SLF  is  envisioned  as  an  integrated  (and 
system  that  will  be  capable  of  generating  a  dense 
:  ••  threat  test  environment  and  simultaneously  monitor  key 
i"  •. meters  of  the  system  being  tested.  It  is  expected  that  this 
::d  it°ly  will  be  part  of  the  EMETF ,  both  physically  and  function- 


ton  d  to  encumber  this  discussion  of  the  test  concept  to 
c  ifferentiate  between  existing  and  new  capabilities  or 


■  ‘-s  the  potential  role  of  SLF  testing  and  the  benefits 
cf  SLF  and  other  modes  of  testing.  SLF  testing  offers 

■  -f  complete  security  and  precise  control  in  developing 
may  be  highly  stressed  (many  simultaneous  but  different 


.  r-  V  *.•; 


levels)  with  the  ability  to  monitor  and  record  data  that 
the  performance  of  the  system  being  tested.  Open  range 
f-  rs  the  advantage  of  producing  test  results  that  often 
i.  In  addition,  such  testing  sometimes  is  the  only  or 
mine  some  tests,  such  as  antenna  pattern  tests.  There 
*■•••  st s  that  may  be  needed,  useful,  or  simply  interesting, 
vilities  will  be  necessary  to  perform  these  tests.  The 
.  !•.  r  st -riding  system  performance  would  be  the  ability  and 


'  1  such,  tests.  We  realize,  however,  that  it  may  be 
f  in  ineialiy  unrealistic  to  perform  all  the  tests  that 
v-  r  amenable,  therefore,  to  endorse  the  concept  of  SLF 
‘  r  •; !  and  comprehensive  characteristics  of  such  tests, 

•  me  ether  complementary  tests  may  be  necessary  on  a 


led,  functional  block  diagram  of  the  overall  SLF 
,v.  S  i  r.c«*  :  t  is  expected  '‘.hat  the  SLF  will  be  a  part 
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Figure  14.  The  functional  SLF  test  concept  and  capability. 
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appears  in  Figure  14.  The  design  concept  does  envision  the  SLF  as  being 
completely  self-contained  and  capable  of  operating  independently  of  the  EMETF 
if  required.  This  same  concept  applies  to  each  of  the  SLF  subsystems.  Each  of 
the  simulator  subsystems  will  have  the  capability  to  operate  independently  of 
the  central  computer  subsystem,  if  required  (by  subsystem  failure  or  location 
of  the  test,  for  example).  Typical  operation  would  be  more  efficient,  however, 
by  utilizing  current  and  projected  capabilities  of  the  EMETF  to  generate 
integrated  scenarios  for  complex  tests  that  involve  a  simulated  tactical 
deployment,  generate  the  scenario  file,  generate  and  ma intain .  required 
parameter  files,  etc.  Specific  examples  of  functions  that  the  EMETF  (with 
current  and/or  expanded  capabilities)  will  be  expected  to  provide  for  SLF  tests 
include  the  following: 

determine  simulated  tactical  deployments  of  personnel, 
equipment,  communications  networks,  and  all  associated  electro¬ 
magnetic  emitters  (COMM  and  Non-COMM)  for  both  friendly  and 
nonfriendly  forces 

determine  (automatically)  realistic  frequency  assignments  for 
the  deployed  emitters 

establish  duty  cycles  for  all  deployed  emitters 

determine  and  assign  the  modulations  and  associated  types  of 
traffic  (voice,  data,  etc.)  to  all  networks  in  the  deployment 

create,  update,  and  maintain  the  emitter  parameter  data  base  for 
use  by  the  SLF  central  computer  subsystem 

generate  integrated  scenarios  (time-ordered  events)  that  will 
include  both  static  and  dynamic  representations  of  the  system 
under  test  and  all  other  emitters  (to  include  location  changes 

and  motion) 

identify  and  cull  out  emitters  from  the  simulated  tactical 
deployment  and  the  integrated  scenario  that  realistically  would 
not  be  detected  by  the  SUT  at  the  applicable  point  in  the 

scenar  io 

refine  the  integrated  scenario  based  on  the  culling  (described 
above;  and  download  this  scenario  to  the  SLF  central  computer 

subsystem 


create  the  individual  SLF  subsystem  scenarios  from  the  refined 
integrated  scenario  for  downloading  to  the  SLF  central  computer 

subsystem 
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create  the  scenario  generation  files  required  by  the  real-time 
processing  software  in  each  subsystem  simulator 

accommodate  adaptive  changes  that  would  override  the  scenario 
generation  files  in  real  time,  based  on  SUT  performance  and  on¬ 
line  EMETF  files/processing  (a  future  function  that  would  be 
very  desireable). 

The  SLF  central  computer  subsystem  is  envisioned  to  provide  at  least  the 

following  functions: 

perform  integrated  test  control  and  provide  test  status 

reduce  and  analyze  test  data  to  satisfy  both  real-time 
(quick-look)  and  post-test  requirements 

uodate  and  manage  the  data  bases  that  contain  the  pretest 
calibration  data  and  the  real-time  test  data 

g-.-r.orate  the  integrated  scenario  for  the  entire  SLF  test, 
including  all  of  the  time-ordered  events  for  the  SLF  subsystems 

generate  the  individual  simulator  subsystem  scenarios,  from  the 
integrated  scenario,  in  the  proper  time  sequence 

create,  update,  and  maintain  the  emitter  parameter  data  base  for 
the  SLF  (the  individual  subsystem  simulator  emitter  parameter 
files  could  be  subsets  of  this  data  base) 

create,  update,  and  maintain  a  message  file  data  base 

create  tne  individual  subsystem  simulator  scenario  generation 
files  for  downloading  to  the  individual  simulator  control 
processors  for  use  by  the  real-time  processing  software  (these 
files  would  include  the  emitter  parameters,  emitter  location/ 
motion  requirements,  messages,  and  other  data  necessary  for  the 
■"eal-time  processing  software  in  the  correct  time  sequence) 


manage,  schedule,  etc.,  the  SLF  program. 


The  prim 3 '-y  function  of  the  control  processor  in  each  of  the  simulator 
nun  systems  wi  :  !  be  to  support  the  real-time  processing  software  for  the 
.;ur  r.y st "ion  -'•Titr.:.  1  processor  may  support  additional  software  functions, 

■  itr.-r  ■’  in  •»  v*>rround  mode,  to  provide  a  stand-alone  capability 

f  o'  :  s’!  a,  ’’Ci-r  at  i  ng  scenario  files  for  real-time  processing, 

"•••>•  r  :  no  ,  :  1  :•  •,  -n  1  maintaining  emitter  parameter  files,  and  creating, 

.piitin-',  !"/:  -  i  into  i  n  i  ng  :n-> ige  files.  As  has  been  noted,  these  off-line  or 
i  ;  >ft«nr-  functions  may  be  performed  in  the  SLF  central  computer 

*•  •  n  1  v  the  scenario  generation  files  necessary  for  the 
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real-time  processing  software  to  function  being  downloaded  to  the  appropriate 
subsystem  control  processor  prior  to  the  test. 

The  real-time  functions  and  division  of  work  in  a  subsystem  simulator  may 
be  understood  better  by  creating  an  example,  using  the  Non-COMM  Threat 
Simulator.  The  control  processor/software  tasks  would  include: 

reading  scenario  events 

maintaining  emitter  and  receiver  locations 
representing  emitter  and  receiver  movements 
reading  emitter  data 

writing  emitter  data  to  the  digital  generator. 

The  digital  generator  tasks  would  include: 

receiving  emitter  data  from  the  computer 
converting  emitter  data  into  pulse  commands 

representing  dynamic  changes  in  PRF/PRI,  scan  rates,  and  carrier 
frequencies  on  a  pulse-by-pulse  basis  for  each  emitter 

providing  these  pulse  commands  to  the  rf  management  section. 

The  rf  management  tasks  would  include: 

allocating  pulse  commands  to  available  rf  sources 

controlling  rf  sources  to  generate  the  pulses 

distributing  the  rf  signals. 

A  message-generating  device  known  as  the  Test  Item  Stimulator  (TIS) 
'current  EMETF  capability)  will  become  the  control  processor  of  the  Functional 
Systems  Simulator.  An  advanced  version  of  the  TEWES  (see  Section  2.1.1)  is 
being  purchased  by  USAEPG  and,  in  effect,  is  expected  to  become  the  Non-COMM 
Threat  Simulator.  The  range  of  frequency  coverage  will  be  500  MHz  to  18  GHz. 
Tne  V'MM  Threat  Simulator  will  be  a  new  capability.  Modulation  capabilities 
rf  source  capabilities  for  this  simulator  are  readily  available.  The 
minimum  radio  frequency  for  COMM  Threat  Simulator  operation  is  not  stated, 
however  the  upper  limit  is  500  MHz.  The  coupling  of  rf  energy  from  the  rf 
’  oar  :es  to  the  SOT  is  a  problem  with  substantial  challenge  because  of  the 
'hr-'r  w-! v “lengths  associated  with  the  operating  frequencies  for  typical  COMM 


3h 


systems  -and  equipments,  which  translate  into  very  large  test  enclosures  if 
normal  far-field  coupling  of  the  rf  energy  is  presumed.  One  consideration  in 
this  regard  is  that  the  rf  sources  and  the  rf  distribution  function  may  be 
located  physically  in  the  associated  Mobile  SUT  Interface  Unit  (possibly  for 
the  Non- COMM  Threat  Simulator,  as  well)  with  direct  coupling  of  rf  energy 
rather  than  antenna-to-antenna  coupling  of  radiated  energy.  Of  course,  the 
interface  units  to  systems  under  test,  the  central  computer  and  test  control 
station,  an:  the  test  data  monitoring  subsystem  also  are  new  capabilities  that 
w i  1 1  n  t  j  be  developed  for  the  SLF  or  adapted  from  existing  genera  1-purpose 


STRUCTURED  APPROACH  TO  PERFORMANCE  DESCRIPTION 
t";.s  section  presents  a  structured  approach  to  the  problem  of  selecting 
•  ■  • :  ,-r a"'’-' ter s  to  describe  the  performance  of  the  various  systems  that  may 

t jsing  the  SLF.  The  proposed  approach  is  not  theoretical  or  unproven; 
i  •  ■,  in  fact,  widely  used  by  national  and  international  standards  organize- 

t  ;  responsible  for  defining  performance  measures  and  objectives.1  The 
i  *r:  provides  the  step-by-step  procedures  required  to  ensure  that  the  set 
:  per f ormane*  parameters  selected  to  characterize  a  system  is  complete, 

>  f  f  i  i-'-n t ,  and  measurable. 

P>'  o  »■ L  c us  p  '.rarneter  development  studies  have  been  approached  from  two 
-  .  p  rr spec ti Voss  that  of  the  user  and  that  of  the  engineer  or  designer, 

i’-e  r ‘“.hiv-s  of  parameter  development  are  fundamentally  different  in  the  two 
"i,vs,  i r.  :  the  appropriate  parameter  sets  differ  correspondingly.  User- 
•’ ienfoi  perf crmance  parameters  are  intended  to  be  applied  in  two  principal 
ways:  '  !  in  specifying  the  performance  requirements  for  a  system  that  is  yet 

'  .  :•  selected  or  designed  and  (?)  in  comparing  performance  among  systems.  To 
v;  prop-i  ate  for  these  -applications,  the  user-oriented  parameters  should 
f  :  .i  ■  user -perceived  performance  effects,  rather  than  the  in  causes 

'  .  ay  stem  and  ' '  not  depend,  in  their  detailed  definition,  on 

1  is  •  i ng  ;s-:  i  hv  -at  leas1-  two  Study  Groups  of  the  International 

:•••*'  «;  •  :  Telephone  Consultative  Committee  (CCITT),  an  organ  of  the 

i  >•  i  -o  . .  Te  1  o  oommun  i  cat  ion  Union  (ITU).  For  example,  Study  Group  VII  is 

:  •  .•••'.  i  cuality  of  service  parameters  for  communications  via  public  data 
i  ’.-'win.*  t.o-  structured  -approach,  and,  Study  G^oup  XVIII  has  adopted 
*•  ••  =  n:it*'i-.;  framework  for  the  development  of  Integrated  Services  Digital 

■;  .  -r  f  3nm.ar.ee  parameters. 


Such  parameters  may  be 


;  s  ;•  out  tr.e  system's  internal  lesign. 

. . *»  system  independent. 

-riented  performance  parameters  are  intended  to  be  used  in  the 
-if;  ;•«*  r;  of  individual  system  components  and  in  relating  such  component 

••  ■  s’- :  r  =  wit-  the  end-to-end  performance  objectives.  To  be  appropriate 

f  •••  it  :  1  icat  ions,  the  engineering-oriented  parameters  should  (1)  be 

mi  specifically  tailored  to,  the  internal  architecture  of  the 
•  be  usef ji  in  identifying  the  causes  of  user-perceived  perform- 

; -or.tr  is"  to  the  user-oriented  performance  parameters,  these 
■  r.g-  ner.tei  parameters  are  system  specific. 

I1:-  s  ■  ’  •;  i  ;ppl  icat  ion,  identified  above,  for  use  of  engineer  ing-oriented 
■  .11  te  r.-'ted  part icular ly .  Vost  system  specifications  depend 

•  *•  of  -T.'ineer  ing-oriented  parameters  to  define  required 

,  «  .  •  •  •••  imp  1 1  tit'  «xpe  *♦  at  ion  that  the  system  will  satisfy 

.  •  ’  •  . :  m-nt  s  •  f  ‘n-'-se  eng  ineer  i  r.g- a-  i  »nt  ed  parameter  specifications 

'  -  •  "'A:v'd  will  ne«*  i  to  perform  some  eng  ineer ing- 

•••  '•  . •  --r.-r.*  •  in  -'d.--  *  •  understand  some  of  the 


i  3*  - ' > 


-■  ip«-i  in  t.nis  study  will  be 

-t’.  *>  f  systems  (existing  or  under 
.  .  the  time  to  detect  and 

,  f-i^o,  f  mused  on  the  develop- 

■  *  -m  t'.  parameters.  However,  it 
tv  ar.-m  engineering  applications 
.  i  ■  •  e  -  i  s  t  i  o  s  and  operating 

in  l«te  'ti  ng  interference.  A 
.  is  that  it  is  useful  in 

i  *.  /  -5-piify  th*1  t.asg  of  relating 
i  p a- am.-tor-s.  a  m’ear  definition  of 
3/st->m  lesign.  The  proposed 
— wr.e*h-'r  if  t*'.  used  in  ccmnuni- 
Iron ; c  survei 1 lan-e,  or  many  other 

oriented  and  the  engineering-oriented 
:  p- inary  parameter s,  which  describe 

n  r m  =  ’  e^at  ion  ;  a n d  secondary 


tne 


describe  the  frequency  and  duration  of  ou’.ages  ( i.e. , 
Lability").  Tne  latter  parameters  are  termed  "secondary"  to 
fact  that  their  values  are  derived  from  observed  values  for  the 
tens,  rather  t.nan  from  direct  observations  of  the  system.  The 
•mar. ?o  parameters  are  developed  in  four  major  steps:  system 

:t:cn,  function  definition,  performance  outcome  analysis,  and 
■tier..  Thes>'-  steps  are  illustrated  in  Figure  ’5  and  described  in 
'elow.  Section  4.4.!  describes  the  proposed  method 
-  a-'  iarv  (  availability)  parameters. 

System  Interface  Definition 

sr*>p  in  developing  parameters  to  describe  the  performance  of  a 
t.-.-  '-ern’o  interfaces  or  boundaries.  This  step  should, 
*  inguish  »,Viat  is  inside  the  system  from  what  is  outside  the 

•  iden*  ify  tne  normal  or  intended  interactions  between  the 
eriv  :ro:.m-*nf  .  Interface  definition  is  stra  ightforward  in  many 

i"\  a  :  ratable  field  radio  set  has  clearly  defined  electrical 
«r:.i  i.-ies,  and  each  boundary  has  an  associated  "protocol,"  or  set 
r:  *•.«  interactions  across  it.  In  other  cases,  the  defini- 
-.•••  tubes  "tne  system"  requires  careful  thought.  For  example, 
rf  ic>.  *  :  t  p  icket  switching  system  involves  separate  physical 
;nd  racke*  level  protocols  that  are  implemented  in  computer 
*'*  w i‘  - ■  a c w  end  of  3  physical  access  line.  Experts  often 
w-:-'-r<-'  "tne  racket  switching  system”  ends  and  "the  user"  begins. 
Lncl--  definition  of  system  boundaries  is  right  for  all  perform- 
t  apr 1 i;  utions:  the  choice  depends  on  the  focus  and  objectives  of 
:*  i  -d“ f  -.r.ed  a 3  "the  system"  in  one  study  might  be  a  subsystem 
r'Lv,  i;.w  of  a  system  nay  be  a  single  entity  (e.g. ,  the 

-r  a  ’.lection  of  entities  (e.g.,  the  operator  and  the 
•  >  :  >  n-i :  ng  or.  where  the  system  (or  subsystem)  boundaries 

•  .  •  •  •  ;  ‘  -  s>;  looting  a  system's  boundaries?  This  depends, 

•  ;  .  -  f  —  mv aaures  of  interest.  In  many  instances  the 


;n>-  1  to  describe  a  collection  of  user  and 
■  ■ive  rv ; cos  from  a  subsystem. 


■  ■ .  a.vv  -  c.V  ■ . .  .*•  -  .  • .  - 
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’.nt-vest  is  the  aser/systen  interface  (sometimes  called  the 
.ter face  .  The  user,  however,  may  be  a  human  operator  or  an 

:  rogram.  irfher-*  t.ne  end  user  is  the  human  operator,  the  user/system 
3  defined  as  the  physical  interface  between  the  operator  and  the 
.  ,  the  keyboard  *  or  the  operator’s  medium  of  inputing  the  terminal 
r.  ca'-d  or  taco).  Outputs  may  be  visual  displays  or  recorded 
n*..o  trie  end  u.  ->r  is  an  application  program,  the  user 'system  inter- 
:  :  :  ’  -■>  t'n-  functional  interface  between  that  program  and  the 


1  -•  3-;— veil  lance  system,  various  interfaces  are  indicated  in 
igure  n.-pi'ts  the  surveillance  systems  in  a  field  test 
in*  a s  obtained  from  a  remote  site  for  use  in  locating 
Th*  -'.'-use*'  interface  is  between  the  human  and  a  display 
i  ♦•••'face  is  between  the  application  program  and  the 
•.  *  .••*■''♦  ''ol  system  which  controls  the  generation  of  signals 
-a*  :  Performance  parameters  are  measured  by  recording 

■ron  v>  'f  signals  or  events  that  occur  at  these  interfaces. 

w  .  1  1  be  measured  as  the  time  between  the  event  that 
i '  c.'ne-'itcr  and  the  subsequent  »vent  that  turned  on  the 
•:*-.'.  detection.  ~>ther  display  times  indicating  signal 
t i f  i ' a* i on,  and  locati m  woul  d  be  measured  over  many 
-  i  a  *  i  v- ■  1  /  large  sample  of  data  could  b°  obtained.  These 
v-.  v  ‘  h-  basi'  units  of  observation  from  which  overall 
am  ■'♦.•■'I";  will  be  defined.  Note  that  other  interfaces  can 
•  rat  oh  sy stem  performance  parameters  can  also  be  measured. 
' '"mur  i  *  a” i on  data  link  performance  can  be  independently 
.--■■■.to  occurring  at  the  input/output  interfaces.  These 
,;v.  •  *•  ■•;  Iced  clocks  to  timr‘  and  record  the  occurrence  of 


„  n  :  t  i  ••'  n  De  f  i  n  i  t  i  o 

•cp  ;lt imat-l  y  refers  t*-.  some  particular 
o  •  cor. d  ;>t ep.  in  developing  system  performance 
tne  specif i-  function,  or  set  of  functions, 
T:e.  •■•  m-iv  comer i all  functions  thr  system 
*b. *•'■•'*  •.*•■»  s  af  a  particular  study. 
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r,-:-i:res  a  ?l*-er  definition  of  the  term  "function"  as  applied  to 
'■n  of  system  performance.  ft  useful  starting  point  is  the 
b:  fin  it  ior.;  a  function  is  a  set  of  ordered  pairs  of  elements  (x,y) 
r.  i  tr.ly  cr.e  value  y  corresponds  to  each  possible  argument  x. 

•  ’.‘sir  1  e  arguments  is  the  domain  of  the  function;  the  set  of 

•  i  ?  “re  of  the  function.  The  sets  X  and  Y  may  include  all 

:  •  ■  o  * :  t  r. :  the  defined  range  and  domain  (thereby  defining  a 

.  „  ..  ,y  include  only  certain  selected  elements  (thereby 
f  ,n-“.  1  .  The  mathematical  definition  of  a  function  is 

•  .r  -  ’ " ,  ;  ?h  portrays  the  function  as  a  machine  that  converts 
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ii  -.  •••.  t  of  a  function  can  be  applied  very  naturally  to 
:/  v. >:■-  f  r  tiers  for  the  purpose  of  performance  description. 
•’  ••  :  is  a  machine  that  performs  a  set  of  specified 

•  \  :•  are  the  function  arguments.  The  system’s  outputs 

:i  . .  .us,  l  system,  function  is  a  set  of  ordered  pairs  of 

.■■.vs:  ••  ;t puts  such  that  one  and  only  one  expected  or 

r-  :  '  reds  to  each  possible  input.  Like  mathematical 
:  *  i  '-.s  may  be  continuous  or  discrete.  In  general,  a 

. ;  fi>.  “:'ns  and  may  perform  many  different  functions  on  a 
.a;  •'  f  s  internal  state.  This  "multistate"  aspect  of 

'  .  •  ” -'presented  by  modeling  the  system  as  a  finite  state 

•  by  and  “IcManamon  (1978). 

-  •  •  r  f  defining  system  functions  for  performance 

•:  *t  ate  -..  Each  function  is  defined  by  specifying  one  or 
a:,  i  issooiated  system  outputs.  There  is  a  single  expected 
ut .  If  to-  system  must  be  in  a  particular  internal  state  to 
•.*  st  -,te  (and  the  inputs  required  to  achieve  it)  should 

’.r.i  -  s.  i  ilance  system,  the  primary  functions  are  signal 

•  i>-  i"  •;  -  intercept  signals  from  a  background  of  noise  and 

t*’^  is  at  i  on  ■  i .  e.  ,  measuring  specific  attributes 
■  :  . ;  .  - ,  hearing,  etc.l,  and  identification  and  location 

•  :  •  •  •*<•••  type,  and  position  fixing). 

•  i  . lives  an  evaluation  of  bearing  angle  measurements 

;•  ••  ■•"■■  •  t  ii'.«  *  :  -cm  a  communications  subsystem. 


,  I 


ARGUMENT: 


fy.V.'-'.VVV.V,V  /-/.  /  vv(v,/ . 


Figure  17.  Illustration  of  the  function  concept  as  a  "machine"  that 

converts  arguments  with  domain  X  (inputs)  into  values  with 
range  Y  (outputs) . 


3  different  set  or  functions  pertains  to  different  systems.  For  the 
'at  Lons  link,  three  major  functions  have  been  defined  by  Federal 
’  !  ’03  3  L  GSA ,  1985  ).  First  a  path  must  be  setup  between  the 
'"•.tors.  This  connecting  process  is  known  as  the  access  function.  Then 
"'or  mat  ion  must  be  exchanged  across  this  path — the  transfer  function. 
t\-  ■cnnection  is  broken  so  other  connections  can  be  made.  This  is  the 
"t  •••r  release  function.  Examples  for  this  and  other  specific 


of  functions  that  characterize  the  system  again  depends 


’•a  description  objectives.  Thus,  there  is  no  need  to 
't  pertinent  functions  are  required.  The  important 
“Leeting  functions  is  that  the  input/output  events  be 
appropriate  interfaces.  Two  general  classes  of 
•••red.  The  first  is  when  a  single,  unique  input  is 
.t  —  for  example,  when  an  emitter  is  detected  and 
The  second  class  is  when  more  than  one  input  is 
’--for  example,  when  a  location  is  determined  based 
wrings. 


Performance  Outcome  Definition 

;  in  ;■  •  -"romance  parameter  development  is  to  specify,  for 
!  r-'t  of  distinct,  possible  outcomes  that  may  be  observed 
perfc'rr.ano  trial  attempt.  Three  possible  outcomes  can 


The  function  is  completed  within  a  specified 
tir:-:  and  the  result  or  outcome  is  within  the 


,  The  function  is  completed  within  the 
,formar.-,e  time,  but  the  result  or  outcome  is 


:t  completed  within  a  specific; 


r'd  into  a  sample  spue  is  illustrated 
;-pe 't  i  ve  l  y  ,  w'th  the  three  general 
;u‘ eit  1  y  expressed  Pv  system  users 
•  •  .r  i  ’V,  -*n1  .;e;y*r.:i  ib  i  1  i  t  v .  Thus,  if  a 


•r. 
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Table  2.  Primary  Functions  of  Specific  Systems 


Navigation/ 

Electronic 

Communication 

Timing 

Remote  Sensing 

Surveillance 

Access 

Acquisition 

Detection 

i  Dprh 

and 

and 

Signal 

>■> 

Estab  lishmer.t ) 

Phase  Lock 

Track ing 

Detection 

F 

T  r  3  nsf  o  r* 

Cycle  Matching 

Range 

-  In  for mat  ion 

and 

and 

Signal 

r 

Exchange) 

Synchronization 

Doppler 

Characterization 

- 

Position  Fixing 

Emitter 

1  1  ser.g  igement 

and 

Target 

Identification 

’ 

r-'-.th  release) 

Guidance 

Identif ication 

and  Location 

•m  erf  ;rms  as  intended,  the  user's  concern  is  with  speed.  This  is 
v  - :  zy  the  system's  delay  in  performing  the  function  on  an  individual 

t a--  rate  at  which  it  can  perform  the  function  in  a  series  of  repeated 
If  tr.e  system  performs  incorrectly,  the  user's  concern  is  with 

i  — the  closeness  of  the  output  to  the  intended  value.  This  is  often 

sse- :  ;  r.  terms  of  an  error  probability.  Similarly,  nonper formance  outcomes 
msec:  ited  with  a  user's  concern  with  dependability.  Such  outcomes  may  be 
wy  ;i  probability  of  function  nonperformance  within  the  specified 

As  an  -.-x  ample,  the  surveillance  system  is  used  again.  Emitter  location  is 
sir*  !  output.  Obviously,  if  this  is  to  be  useful  target  information,  the 
performance  is  judged  on  how  fast  and  how  accurately  the  location  is 
.  If  the  location  is  incorrect,  repeated  trials  may  improve  the  precision 
or/  :*■  --r-'-are  beyond  acceptable  limits  the  time  required  to  locate  the 
.  I*’  the  emitter  is  undetectable,  a  case  of  nonper  formance,  then 

t  ir.g  metr.ur--r.entr>  may  not  suffice.  These  outcomes  depend  on  the 

m  tv-  ier.ired  performance  description  objectives. 

nnouid  V*  defined  in  terms  that  are  easily  understood  (e.g. , 
‘  r  t .  ,  f  i  i  detect  ion  probability,  nondetection  probability). 

■'I  it 


A. 


Successful  performance  is  achieved  if  it  meets  specified  criteria  for  the 
maximum  time  specified  to  achieve  the  outcome. 

4 .  Parameter  Selection 

Parameter  selection  is  the  fourth  step  in  the  logical  four-step  process 
for  developing  performance  parameters  for  any  system.  Primary  parameters, 
described  in  Section  A.H.i,  are  required  to  characterize  performance  under 

expected  (normal)  operating  conditions.  Secondary  parameters,  described  in 
Section  ^.9.2,  define  performance  over  the  long  term.  We  apply  this  four-step 
process  to  the  development  of  functional  performance  parameters  for  two 
specific  EWI  systems  in  Section  5.  This  structured  approach  applied  to  the 

development  of  performance  parameters  for  digital  communication  systems  is 
defined  in  Federal  Standard  (FS)  1033  (GSA,  1985)  and  American  National 
Standard  (AN'S)  X3.102  (ANSI,  1983)  and  explained  in  some  detail  in  a  report  by 
Seitz  and  Grubb  (1983).  A  summary  of  the  approach  applied  to  digital 
communication  systems  is  given  in  Appendix  B  of  this  report. 

Primary  Parameters 

The  final  step  in  parameter  development  is  to  select  and  define  particular 
r  tr a meters  to  describe  the  performance  of  the  system  relative  to  each  specified 
fun 'tier  and  outcome.  The  parameters  will  normally  be  random  variables  defined 

n  -n  outcome  sample  space  (and  associated  performance  time  distribution).  The 

r or  ;m-  tors  selected  will,  of  course,  depend  on  the  system,  the  functions,  and 
the  outcomes  considered.  The  set  of  selected  parameters  should  have  the 
following  general  attributes: 

lompleteness.  As  a  set,  the  selected  performance  parameters  should 
express  all  performance  attributes  of  major  significance  to  the 
study.  Parameters  should  reliably  reflect  actual  performance  over 

the  full  range  of  possible  values. 

Efficiency .  The  selected  parameters  should  be  as  few  in  number  and 
simply  defined  as  is  possible,  consistent  with  the  study  object- 
i.'os.  The  parameters  should  be  nonover lapping — each  should  express  a 
i i  f  f‘-r--r.‘  aspect  of  performance.  The  parameters  should  be  directly 
r- ar.t  to  the 3f-  who  wi  1 1  use  them. 

"■ 1  i  1  j  ty  ,  The  performance  parameter  definitions  should  be  based 

ri.-r.als  or  ey-r.ts  that  are  directly  observable  at  the  system's 
ir.terf  i'>e:;.  The-  parameters  should  be  measurable  during  normal  system 
'  ;  ■  it : '  s  under  various  test  scenarios.  The  parameter  definitions 


shou.l  ce  mathemat  compatible  with  statistical  estimation 

toe  hr:  i  tp^rs  to  enable  the  precision  of  parameter  estimates  to  be 

quart  it  at  ivelv  stated. 


x ample,  consider  again  the  performance  of  an  electronic 
system.  The  first  major  function  of  such  a  system  is  to  detect 
of  a  signal  in  a  background  of  noise  and  interference.  A  basic 
■  te;t  ion  function  and  the  associated  outcome  possibilities  are 
ri.-ure  '•  9 .  The  function  input  is  either  a  signal,  S,  or  no 
.;tpu‘  ran  be  either  an  indication  of  signal  detection,  D,  or 
!  .  Thus,  there  are  four  possibilities  shown  in  the  matrix  in 


ies  are  as  follows: 

•nt  and  is  detected  within  a  specified  maximum 
The  relevant  parameter  is  the  detection  time, 
i", ere  a  succession  of  signal  detection  trials  is 
■•-tion  rate. 


•Terence  pulse  (or  an  equipment  malfunction)  is 
valid  signal  when  no  signal  is  present.  _The 
>r  is  the  false  detection’  probability,  P(D|S). 

3k en  for  a  noise  or  interference  pulse,  or  for 
net  detected  within  the  specified  maximum 
The  relevant  parameter  is  the  nondetection1* 


,  and  no  signal  detection  is  reported  within 
;:.'S»rvat ion  period.  .Since  no  detection  function 
performance  parameter  is  necessary  or 


i* ■>  correspond  exactly  with  the  intended  performance, 
mar.  -  v,  f  c  ^performance  outcome  categories  defined  earlier. 

;  i ■  ;  1  i t  y  is  null  and  is  correctly  excluded  in  parameter 
-  -col  approach.  Outcomes  ,?  and  3  correspond  exactly 

refined  in  statistics. 

-  *  -•  s  imm.ar  ized  in  the  sample  space  diagram  of 

•  .■  uld,  of  course,  be  defined;  but  with  the 


e  alarm  in  radar  detection  theory. 

...  ‘-.n-.f: ior  in  rad  a**  detection  theory.  The 
io:.r  Jor'  i  ate,  sin~o  it  is  easier  to  interpret. 
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exception  of  the  need  for  availability  measures,  described  below,  these 
measures  appear  to  satisfy  the  completeness,  efficiency,  and  measurability 
criteria  defined  earlier.  Presentation  of  the  performance  parameters  in  an 
outcome  sample  space  emphasizes  the  relationships  among  them  and  can  clarify 
the  performance  implications  of  system  design  decisions.  For  example,  non- 
deteotion  ir.  many  systems  is  a  function  of  a  bias  setting  or  detection 
threshold.  In  general,  nondetection  probability  can  be  traded  for  false 
detection  probability  by  varying  the  threshold  setting. 


jr. 


,-.2  Secondary  Parameters 

Primary  parameters  of  the  type  defined  above  can  provide  a  very  detailed 
ascription  of  performance  during  periods  of  normal  system  operation,  but  they 
■  n't  meet  the  need  for  quantitative  description  of  the  frequency  and  duration 
'  outages.  A  separate,  typically  smaller,  set  of  "secondary"  parameters  can 
-  defm-cd  to  meet  that  need.  The  secondary  parameters  describe  system 
?r  romance  from  the  more  general,  macroscopic  point  of  view  traditionally 
w;  ste  i  with  the  concept  of  availability. 

-igure  -’1  illustrates  the  method  for  developing  the  secondary  parameters, 
it  UV3  are  defined  by  comparing  values  for  selected  primary  parameters  with 
if  led  outage  thresholds  during  successive  performance  periods.  A  defined 
.•  a  ;  1  a  :>  i  1  :  t  y  function  (e.g.,  inclusive  or)  maps  threshold  violations  into 
■-■•  a.  Key  availability  definition  issues  are  the  selected  primary  param- 
•?rs,  the  outage  thresholds,  the  performance  period(s),  and  the  availability 
motion.  The  proposed  approach  reflects  the  view  that  an  outage  is  an 
ir  deniable  degradation  in  system  performance  that  may  or  may  not  involve  a 
■ril  service  cutoff  or  equipment  "crash." 

.?  1  illustrates  a  simple  two-state  availability  model  and  the 
;  '  i  p  irameters.  h’nder  the  (common)  exponential  assumption,  transitions 


:oe  available  and  unavailable  states  are  represented  by  the  failure 


?  res torsi  rate  u*-or  equivalently,  by  their  reciprocals,  the  Mean 
■r.  Failures  'MTrtF;  and  Mean  Time  To  Repair  (MTTR).  The  availability 
ivii  list  lily  may  be  calculated  directly  from  these  quantities  as 
Th-'-  parameters  \ ,  u,  MTRF,  MTTR,  A,  and  tJ  are  all 
rime tors.  Specifying  any  two  nonreciprocal  parameters 
.;•<  arameters  may  be  calculated  from  a  single  related 
r  ~  rah  i  1  i  ty  ,  in  the  special  case  where  (1)  the 


'■e  :  irure. 

. v a  i  1  at  1 1 i tv 


.  .  .0000101001  100.  . 

Availability  states  f  f 

during  successive  - — 

performance  periods 

Figure  21.  Determination  of  availability  states. 
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Figure  22.  Definitions  for  availability  and  unavailability. 


oer  ;eas 


equal  duration,  and  (2)  successive  outages  are 


:ier.t.  Additional  parameters  may  be  defined  to  describe  the  dependence 
i:-;V’rirg"  cf  outages  if  the  exponential  assumption  is  inappropr iate, 

;  ar.  example,  assume  that  the  surveillance  system  exhibits  a  high  false 
or.  probability  that  exceeds  the  user  acceptability  threshold  due  to  an 
•r  oita  setting  or  some  component  failure.  This  acceptability  threshold 
•writ  arbitrary,  but  when  exceeded,  it  means,  to  the  user,  that  the 


iv a  i .  abl* 


for  use;  i.e.,  the  system  is  in  an  outage  state  until 


■r  repairs  ar->  made.  Of  course  the  detection  function  failure  is 
several  functions  that  impact  outage  in  a  surveillance  type  system. 

loss  of  bearing  measurement  could  preclude  locating  the  emitters 
leg  tr>-m— the  ultimate  system  objective.  The  same  processes  are 
.  t  i  f  v  mi  define  secondary  parameters— namely  interface  definitions, 
•fin  if  ion,  performance  outcome,  and  parameter  selection.  The 
f met  ion  may  be  defined  as  the  probability  that  the  system  will  be 
•  : rg  at  ate  at  time,  t,  during  the  total  mission  time,  T.  A  relia- 
sometimes  used  that  is  the  probability  that  a  system  will 
ic  >?;.•*■  able  thresholds  throughout  the  total  mission  time  (over  the 
il  core  to  T‘.  The  user  is  particularly  interested  in  the  avail- 
system  is  no  be  used  and  may  not  be  particularly  interested  in 
times  the  system  has  failed  and  been  repaired  before.  Both 
: * .  1  ”el lability  depend  on  thresholds  chosen  for  some  or  all  of  the 
:  paramete"s  and  are  not  necessarily  the  result  of  total 


PrFF  RM.AN'E  DESCRIPTIONS  FOR  TYPICAL  EWI  SYSTEMS 
ve loped  a  structured  approach  to  the  problem  of  selecting  sets 
1-  ih*  t;*.e  performance  of  the  various  systems  that  may  be 
.'LI  .  Tnis  section  applies  that  structured  approach  to  two 
items.  rst ,  a  description  of  each  system  is  given  in 

•  in  section  5.?,  the  structured  methods  described  in 
,r  i-'h  • .  -  re  applied  to  ( 1 )  define  system  interfaces  or 

efir.-  fan -t  i  .ns  that  describe  performance  that  is  of  interest, 
it  the  system  interfaces,  (3)  specify  for  each 
f  ;  '1'  1  •  '•  iteomes,  -and  (-'i'  select  and  define  parameters  to 


describe  the  system  performance  of  interest  relative  to  each  defined  function 
and  outcome. 


5.1  System  Descriptions 

Two  electronic  surveillance  systems  have  been  selected  for  this 
methodology  development  study.  Functionally,  these  systems  are  quite  similar 
'though  not  identical),  but  there  is  a  great  deal  of  difference  in  the  complex¬ 
ity  of  tr.ese  two  systems.  The  simpler  system  is  a  ground-based  system  that 
i  lent i Ties  unfriendly  emitters  and  determines  the  directions  of  arrival  (true 
te  trir.gs'.  for  signals  from  those  emitters.  The  complex  system  consists  of 
several  airborne  subsystems,  connected  by  wideband  data  links  with  a  ground 
:  r  i  -nalysis  subsystem,  that  perform  the  same  functions  of  unfriendly  emitter 
:  i--r*  : float  ions  and  determinations  of  lines  of  bearing  to  these  emitters.  It 
•  :  -  performs  the  additional  function  of  using  the  LOB  data  to  calculate  the 
tnpnic’il  locations  of  these  unfriendly  emitters.  A  description  of  the 
;  ' :  1  •> r  system  <nown  as  the  AN/MSQ-103  Special-Purpose  Receiver  Set  (or 
T-:  AM  PACK  Assembly)  is  given  in  Section  5.1.1.  A  description  of  the  more 
om:  '.ox  system,  known  as  the  Advanced  QUICK  LOOK  System,  is  given  in 
:o-tion  5.1.2.  These  descriptions  may  not  represent  the  latest  system  config- 
it  i ms ,  or  details  may  be  omitted  so  that  the  system  descriptions  may  remain 
;n classified.  However,  the  descriptions  are  adequate  for  this  development  of 
*•  r-  m-*  mi  elegy  for  the  SLF . 

"he  AN/MSQ-103  Special-Purpose  Receiver  Set  (TEAMPACK  ESM) 

As  defined  by  the  Bunker  Ramo  Corporation  (1979)  for  the  U.S.  Army  Signals 
Warfare  Laboratory,  the  AN/MSQ-103  Assembly  (TEAMPACK)  is  a  ground-based 
vehicle-mounted),  electronic  warfare  support  measures  (ESM)  intercept  and 
etior.  of  arrival  (DOA)  system  for  identifying  and  locating  threat  Non-COMM 
systems.  The  system  includes  a  permanently  mounted  (on  the  vehicle),  erectable 
•interns  mast  with  heads  that  contain  two  omnidirectional  antennas,  three 
•  iv-r  V>ads  (covering  six  contiguous  bands)  that  each  include  two  DF 
••  t  i  -•  r  finding)  antennas,  a  masthead  switch  (for  band  selection  and 
f  ,  a  three-way  combiner,  and  a  shaft  encoder.  Three  frequency  synthe- 

i  ,  a  frequency  control  unit,  a  pulse  train  separator  (PTS),  a  video 
:  v  v  -r  .nit  7 DM ) ,  an  i n.d icator-processor  unit  (IPU),  a  teleprinter,  and  an 
•  ••  ",  "YE- •  a  neogai’  are  rack-mounted  in  the  vehicle.  Separate  voice 
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communication  equipment  that  includes  an  AN/VRC-46  radio  set  and  a  TSEC/KY-38 
encryption  unit,  along  with  the  necessary  power  supplies  for  all  of  the 
electronic  equipm-nt  noted  above,  also  are  mounted  in  the  vehicle.  (An  rf  test 
set  and  an  azimuth  gyro  survey  instrument  are  included  as  part  of  the 


•  .  t  i  0  r. 


.3  performed  using  the  omnidirectional  receivers.  The 


•a  ;bt  air.ed  includes: 


alarm 


(ting  frequency 


fidth  (PW1 


se-repet if .on  frequency  (PRF). 

arrival  (or  true  bearing)  measurements  are  accomplished  using  a 
direction-finding  receiving  system  that  includes  a  two-antenna 
r  receiver  that  provides  accurate  but  ambiguous  bearing  informa- 
s-',  amplitude-difference,  direction-finding,  two-channel  receivers 
omoiguity  from  the  interferometer  receiver  data.  A  functional 
f  the  system  is  shown  in  Figure  23. 

••nr.-.i  system  functions  for  the  AN/MSQ-103  Special-Purpose  Receiver 


il  detection  and  the  determination  of — 


?quency 


se  width 


.  ilse-repet i t ion  f ^equency 


true  bearing  (or  direction  of  arrival). 


•.•rir.g-oriented  functions  that  entail  testing  and  validating  signal 
t-iey  are  used  to  provide  information  to  the  user  include: 


e' . i  n.'  for  adequate  power  level 

.  r — :  i  n  ir.g  that  signals  are  within  the  proper  field  of  view 
i*'"erm  in  i  nr  that  signals  are  within  the  selected  frequency 
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the  capability  to  sort  six  different,  validated  signals  that 
simultaneously  occur  within  the  IF  pass  band  and  perform  the 
user-or iented  functions  for  each. 

g ineer i ng-or i en ted  functions,  of  course,  are  essential  to  the 
nted  functions  noted  above. 

en  characteristics  that  relate  to  functional  performance  and  that  will 
■>  measured  using  the  capabilities  of  the  IWS  include  the  following: 

discrete  and  scanning  frequency  selection  capabilities 

in  :: vid-ial  receiver  and  receiver  set  sensitivities 

antenna  calibrations  that  establish  field  of  view  (when  combined 
with  receiver  thresholds  and  signal  processing  capabilities  of 

the  system) 

iyna.nio  range 
image  rejection 

effective  bandwidth  of  the  receiving  set. 

■  Advanced  QUICK  LOOK  System 

rrvolopment  Specification  (ERADCOM,  1982)  defines  an  Advanced  QUICK 
to  be  a  number  (three  maximum)  of  airborne,  noncommunications  (Non- 
f-'.nies  intelligence  (ELINT)  systems  that  are  connected  to  a  ground 
■si:  subsystem  via  special  electronic  mission  aircraft  (SEMA)  wideband 
-■  s .  Tr.e  system  is  supported  by  maintenance  subsystems,  for  each 
s  system  and  the  (ground-based)  data  analysis  subsystem,  and  mission 

)  iroorne  subsystem  includes  two  Receiver  Groups  (to  achieve  360° 
■’ ige ; ,  a  Quantizer  and  Controller  Group,  (interface  to  and  airborne 
f '  a  GEM  A  Wideband  Data  Link,  a  Navigation  Unit,  Mission  Test 
,  r.  i  i  Maintenance  Subsystem  as  shown  in  Figure  2A.  Each  Receiver 
• ;  •  •  sf  r w~  sets  of  radio  frequency  processor  units  with  a  low-band, 
i  :  high —  uni  processor  unit,  each  with  an  appropriate  antenna 
t .  Receiver  Group  also  includes  a  direction-finding  (DF) 

f*'--  'IF)  processor  unit,  a  frequency  synthes  izer/ local 

r.  '  ,  :  i  •'ere  5  ver  power  supply.  The  Quantizer  and  Controller 

m.  ;r‘ ic<>  unit  between  each  Receiver  Group  and  the  other 


components  of  the  airborne  subsystem,  the  computer  processor,  a  pilot's  control 
unit,  and  the  system  power  supply. 

The  airborne  subsystem  is  capable  of  both  automatic  and  directed  search 
modes  of  operation.  In  the  automatic  mode,  the  subsystem  tunes  through  the 
frequency  bands  according  to  a  preprogrammed  search  routine  that  is  determined 
prior  to  a  mission,  but  that  may  be  dynamically  reprogrammed  by  the  ELINT 
supervisor  via  the  SEMA  wideband  data  link.  In  the  directed  search  mode,  the 
subsystem  accepts  tuning  commands  from  the  ELINT  supervisor  via  the  wideband 
data  line  to  perform  measurements  at  specified  frequencies. 

The  airborne  subsystem  detection  and  hardware  capabilities,  employing 
phase  interferometer  direction-finding  techniques  along  with  software 
processing  capabilities,  provide  detection  and  identification  of  the  following 
types  of  emitters'  signals: 

continuous  wave  (CW) 

stable  pulse  repetition  frequency  (PRF)5 
’itter  (±2  microseconds  maximum) 


stagger  to  16  levels 
nonperiodic  random  PRF 


Yeouency  hopper 


swept  frequency 


E  A  chirp 
phase  coded 

sub-pulse  frequency  step. 

ic ip ’ e  signal-sorting  parameters  are  angle  of  arrival,  frequency,  time 
va  1,  and  pulse  interval.  Processed  emitter  data  are  sent  via  the 
data  link  to  the  data  analysis  subsystem.  These  data  include  emitter 
7,  puls«  repetition  interval  (PRI),  pulse  width,  emitter  location 
srd  longitude,  time  of  intercept,  and  direction  of  arrival  for  each 


nd  PRI  are  used  interchangeably  in  the  Development  Specification 
",  1  no?)  without  careful  attention  to  the  difference  between  these  data. 


v.WiW( 
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signal,  and  platform  self-location  data  (attitude  and  location,  using  an 
appropriate,  but  undefined,  three-dimensional  coordinate  system). 

The  data  analysis  subsystem  includes  all  processing  capability  necessary 
to  perform  all  correlations  of  data  received  from  up  to  three  airborne  sub¬ 
systems  necessary  to  determine  locations  of  emitters.  Appropriate  operating 
mode  requirements  for  the  airborne  subsystem  are  programed  by  the  ELINT 
supervisor  into  the  data  analysis  subsystem  in  response  to  requests  from  users 
ir.  the  field  for  (unfriendly)  emitter  location  information.  Intercept  data, 
provided  via  the  wideband  data  link,  are  processed.  Following  processing  of 
these  data,  appropriate  messages  are  prepared  by  an  ELINT  operator/analyst, 
approve!  by  the  ELINT  supervisor,  and  forwarded  to  users  as  required.  A  block 
diagram  of  the  data  analysis  subsystem  is  shown  in  Figure  25. 

The  data  analysis  subsystem  provides  control  to  and  receives  data  from  up 
t  ■  three  airborne  subsystems.  The  data  analysis  process  includes  correlations 
■f  intercepted  data  with  stored  information  for  known  emitter  types  and 
locations,  the  calculations  required  to  identify  the  type(s)  and  location(s)  of 
t-  w  emitters  intercepted  during  the  mission,  and  processing  and  storage  of 
..r. correlated  line-of-bearing  data. 

Components  of  the  data  analysis  subsystem  include  (three)  wideband  data 
: ir  -  interfaces,  a  computer  system  (32-bit  computer)  with  console,  three 
in*' or-^ticr.  display  and  control  units,  a  video  hard  copy  unit,  two  disk  storage 
.r.  its,  a  cartridge  tape  recorder  unit,  a  reel-to-reel  tape  recorder  unit 
' '*-tr  ,  and  a  line  printer.  The  information  display  and  control  units 

provilr  interactive  keyboards  with  alphanumeric  and  graphic  displays  for  access 
to  the  computer  and  accommodation  of  data  transfer  between  ELINT  positions. 
These  units  also  provide  digital  display  of  (emitter  operating)  frequency,  PRF 
or  PEI),  PW,  stagger  levels,  jitter  ranges,  latitude  and  longitude  (of  the 
Inter  jeptod  emitter),  military  grid,  semimajor  elliptical  error  of  probability 
'  -  HP  ,  PEP  confidence  factor,  number  of  intercepts,  time  of  intercept,  and 
■ i grv-d  pr  ior  i  t  ies. 

:  it  t  that  are  preprogrammed  into  the  system  and  that  are  measured, 
n.  •;.!*<  :,  and /  or  stored  by  the  airborne  and  data  analysis  subsystems  are 
■.rv  i hi  •••i  into  three  categories:  the  electronic-order-of-battle  (EOB)  cate- 

*  h-’  new  emitter  category,  and  the  uncorrelated  line-of-bearing  (LOB) 
Tar  le  ?  shows  the  system  data  and  the  source  and  disposition  of  data 
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Table  3.  Data  That  Are  Preprogrammed  (P)  into  or  Measured  (M),  Calculated 
(C),  and/or  Stored  (S)  by  the  Advanced  QUICK  LOOK  System 


DATA  C 

Emitter  Identification  (Name) 

Frequency 
PR?  ( or  PRI'* 

PW 

Emitter  Location  Latitude 

Emitter  Location  Longitude 

Time  of  Intercept  (Last  Confirmation) 

Total  Number  of  Intercepts 

Platform  (A/C)  Self-Location  Data* 

-°latform  (A/C)  Attitude  (Heading) 

-Platform  (A/C)  Location  (Using  Appropriate 
Three-Dimensional  Coordinate  System) 

Direction  of  Arrival  (DOA) 

Semi-Major  Axis  of  EEP 

Semi-Minor  Axis  of  EEP 

Orientation  of  Semi-Major  Axis  of  EEP 


EOB 

itegory 

New  Emitter 
Category  ( 

LOB 

lategi 

P 

P.M 

M.S 

M.S 

P.C 

C,S 

C.S 

P.M 

M.S 

M.S 

P.C 

C.S 

— 

P.C 

C.S 

— 

M,(S) 

M.S 

M.S 

M,S 

M.S 

— 

C 

C 

C.S 

♦Provided  by  the  CAROUSEL  IV-E  Inertial  Navigation  Set  (INS) 


Information  on  the  signal  parameters  and  locations  of  target  emitters  that 
nave  been  established  prior  to  a  mission  comprise  the  EOB  category  of  data. 
Each  known  emitter  is  described  with  the  following  information:  emitter 

identification  (name),  frequency,  PRF/PRI,  PW,  location  latitude,  location 
longitude,  time  of  last  intercept  confirmation,  and  total  number  of  intercepts. 

Information  on  the  signal  parameters  and  locations  of  target  emitters  that 
•vim-  seer,  intercepted  during  a  mission  and  that  are  not  in  the  EOB  category 
-  '".prise  the  n«w  emitter  category  of  data.  Data  to  describe  the  newly  inter¬ 
acted  emitters  include  measured  frequency,  calculated  location  latitude, 
'.--ulate  i  location  longitude,  time  of  intercept,  total  number  of  intercepts 


me  fix,  the  senimajor  axis  of  EEP,  the  semiminor  axis  of  EEP,  and 
F  the  semimajor  axis  of  EEP. 

information  on  the  signal  parameters  and  lines  of  bearing  to  the 
arget  emitters  that  cannot  be  correlated  with  the  EOB  category 
he  new  emitter  category  intercept  records  comprise  the  uncorre- 
" -bearing  category  of  data.  These  uncorrelated  1 ine-of-bearing 
ire  following:  measured  frequency,  measured  PW,  calculated 

of  intercept,  aircraft  location  (using  an  appropriate,  but 
re>.-- 1 irons ionnl  coordinate  system)  at  the  time  of  intercept,  and 
irrival  of  the  intercepted  signal. 

vugh  the  Advanced  QUICK  LOOK  System  is  an  airborne  electronic 
system  'with  a  ground-based  data  analysis  subsystem  and  a  connect- 
bund  data  lir.x)  that  is  much  more  complex  than  the  ground-based 
tr.'.y,  very  similar  user-oriented  system  functions  are  performed  by 
E  Li  -ystem.  These  user-oriented  functions  are: 

■  a  i  detection  and  the  determination  of — 


p  |  •,-»»?  M/ 

*  •  -  t  *  ■  J 


—  vetiti  f-equency  (or  PRI) 

■ -.r  i.v,  with  respect  to  the  system  reference 
:  n,  r  direction  of  arrival,  with  respect  to  another 
1  ion  for  example,  an  aircraft's  location) 

. m-  Local : nr. 

■  '  through  (5)  above  are  common  to  both  systems,  though  at 

-he  technologies  for  performing  these  functions  are  different, 
tv  7 E. Ac’ PACE  Assembly  determines  signal  DOA  by  applying  two- 
fercm-.-ter  and  separate  two-channel  received  signal  amplitude 
overcome  direction  ambiguity)  technology,  whereas  the  Advanced 
a*'1  s  phase  inter  f  erometer  direction  finding  technology. 
;  "  -  the  Advanced  QUICK  LOOK  System,  which  has  the 

■  b  conform  tr iangulation  using  multiple  measurements  of 
mine  .-■milter  locations. 

;  number  of  engineering-oriented  functions,  as  yet 
V  ar.-.l  validating  measured  signal  data  before  they  are 
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accepted  as  confirming  E"'B  data,  new  emitter  data,  or  valid  LOB  data.  An 
example  would  be  the  function  of  checking  some  number  of  sets  of  LOB  data  to 
determine  if  the  EEP  is  within  an  acceptable  limit  that  then  would  support  the 
conclusion  that  a  new  emitter  had  been  identified.  Similar  functions  of 
testing  measured  frequency,  measured  pulse  width,  etc.,  to  determine  that 
measured  values  are  within  specified  tolerance  limits,  also  will  be  required  of 
the  system. 

‘•'any  other  ( engineer ing-oriented)  system  characteristics  that  influence 
functional  performance  will  be  important  to  an  initial  determination  that  the 
system  operates  properly  and  can  be  expected  to  provide  satisfactory, 
.ser -perceived  performance.  These  characteristics,  that  will  have  to  be 
m-MSured  using  the  capabilities  of  the  IWS,  include  the  following: 

frequency  rang--  (for  each  receiver  and  the  system) 

acquisition  (detection)  noise  bandwidth 

frequency  resolution 

frequency  accuracy 

signal  detection  bandwidth 

direction  finding  bandwidth 

sensitivity  (low,  mid,  and  high  bands) 

IF  accuracy  (excluding  navigation  errors) 

ivnanic  range 

spurious  rejection 


image  rejection 

pulse  width  characteristics 

•  range 

•  resolution 
^  accuracy 

•  measurement  amplitude 

:  a  l  --’  r-rpet  i  t  i^r.  interval  characterist  ics 

•  rang*1 

•  resolution 

•  a eouraov 


time 


6-’J 


f  «• 


•r.ten.na  character istics 

•  Txz.  imuth  coverage  .per  quadrant  and  total) 

•  polarization 

system  clock  accuracy 

•1MI  requ ir ements  (frequency,  range,  intensity,  modulation), 


c..?  Definitions  of  MOFPs 

vj’-?:  appro  >ch  to  describing  system  performance  from  a  user’s 
v  v-  lcpe.i  in  Section  now  is  applied  in  the  definition  of 
:  ns  f -r  the  TEAMPACK  Assembly  and  the  Advanced  QUICK  LOOK 

ippr  j.a.'h  includes  the  definition  (from  a  usei — oriented  perspec- 
.•n :  output  interfaces,  system  functions,  performance  outcomes, 

:  - n  ar.  1  1-'f inition  of  parameters  that  describe  system  performance 
.•••"vo  i  system  interfaces,  relative  to  each  function  and 


interfaces  for  the  TEAMPACK  Assembly  are  provided  by  the  video 
;  t ,  *"•'  tulse  train  separator,  the  i nd icator-processor  unit,  and  the 
.  7 : : ■  i  ujs  la" a  that  constitute  system  control  are  provided  by  the 

system  "r.rougn  the  data  entry  keypad  and  other  controls  of  the  IPU. 

-cter  is.. at  ion  of  system  operating  status  and  visual  read-out  (LED 
■. -r  a  ■  at  inns  (frequency,  PRF,  PW,  true  bearing,  and  emitter 
r'-e spends  to  stored  characteristics  of  unfriendly  emitters)  for 
•■••titters  is  provided  to  the  user  by  the  IPU.  Primary  power  control 
rem  is  provided  by  the  user  through  the  VDU.  An  audio  monitor  of  a 
Z'.\  1  signal  being  present  is  provided  to  the  user  by  both  the  VDU  and 
rv'a’.iorated  emitter  characteristics  (frequency  and  DOA  data  for  the 
•.  p e  i  ng  processed)  are  provided  by  the  PT-S  to  the  user  using  an  LED 
; :  a.-rt  window  being  used  by  the  PTS  also  is  shown  using  an  LED 

•  in  control  .MPT  or  DISPLAY  measured  data)  is  provided  to  the  PTS 
"p  to  >: a  s-’ts  of  the  interception  and  true  bearing  data  may  be 
■•  -pu"-?”  memory.  These  same  data,  either  as  individual  sets  or 
■  ■■•-.r  rf  no.'  computer’s  mission  memory,  are  provided  in  hard 

.  ,  ,,  -i  r.  «- teleprinter. 

■"•  ■■••  :  v :  •.,-  antennas  denote  the  interface  between  the  system 
:  ••'•  ■  ■•-.•;  •-;•-, ns  '  source /system  interface).  For  purposes  of  SLF 


n i  ent.  uec 


’oss  to  the  "source/system  interface"  may 
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no  realized  by  working  at  the  interface  between  the  control  system  for  SLF 
transmitters  and  the  overall  testing  control  program  (or  application  program) 
as  shown  previously  in  Figure  16.  A  simplified,  functional  block  diagram  of 
the  TEAMPACK  Assembly,  denoting  these  system  interfaces  and  the  information 
’-.nut  is  provided  to  the  system  or  to  the  user,  is  shown  in  Figure  26.  Informa¬ 
tion  shown  (at  the  interfaces)  in  parentheses  denotes  information  provided  by 
tr-:  .s-  r  v:  the  system;  the  remaining  information  is  provided  by  the  system  to 
r n-'  user ,  as  described  earlier. 

In-erfaces  for  the  Advanced  QUICK  LOOK  System  are  denoted  in  the 
1  f  i •->  i  b  1  to k  diagrams  of  the  system  shown  in  Figures  27  and  28.  Figure  27 
i  a i -p 1  if :ed  block  diagram  analogous  in  detail  to  Figure  26  for  the  TEAMPACK 
ly.  Figure  26  incorporates  further  simplification,  but  also  illustrates 
-nit  -he  system  could  be  tested  as  several  separate  subsystems,  namely  each  of 
the  airborne  subsystems  A-l  through  A-N ) ,  associated  data  link  subsystems 
tor  ugh  DL-N  ’ ,  ani  the  data  analysis  subsystem  (DA).  This  methodology 
>-.!v  assumes  testing  of  the  entire  system,  realizing  that  only  one  or  two 
Su'!Sy stems  may  be  used  to  form  "the  system." 

"s»5r  system  interfaces  are  provided  at  the  pilot’s  control  indicator  unit 
•<•  :  gr  '.ur.1 '  information  display  and  control  units.  Inputs  to  the  system 

;■  :  -  -  ’  s  control  indicator  unit  include  power  ON/OFF  control,  quadrant 
*.  f  -  - -.e  ir/e-cept  receivers,  and  initialization  (zeroize)  of  system 
•  ;  -  ;t 3  fr-m  the  system  via  the  pilot's  control  indicator  unit  are 

a  *  ••  *.  :  ■  ".-ati  ar.d  fault  indications. 

*  •  t he  system  through  the  information  display  and  control  units 
-  :  •■■■  p  imm  i  nr  and  all  operational  support  and  control  for  the 

t  ch  systems.  Operating  instructions  include  the  definition  of  scan 

c  :  ,  r  .nlwid-hs,  dwell  times,  revisit  times,  target  priorities,  and 

•.  .  •  -  !■.  i.-  -ent  lata  MV>?ories.  Outputs  from  the  system  via  the  informa- 

•  ;  •  :  i.  ,m i  •  an*'*’-}!  unit  may  be  requested  using  various  sort  criteria  such 

*  -  .  r  v,  F F  :  'har  acter  ist ics,  size  of  the  EEP  major  axis,  and 
-  •  ,  :  s - 1 -  The  data  that  are  displayed,  depending  on  the  sort 

■,  f-  ;.>n.cy,  PW,  PRF/PRI,  stagger  levels,  jitter  ranges, 

,  .••-r.fi  i-c’*-'  f  ichor,  emitter  location  (latitude  and  longitude), 

v.mbor  -f  intercepts,  time  of  each  intercept,  and  assigned 

•  :  -  ;  •  .  '  a*  ,  that  may  be  displayed  are  stored  both  in  the  airborne 

*  ■  - ,  ir  lit/  f  -  ."'000  FOB  records  and  2000  new  emitter  records, 
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Figure  26.  Simplified  functional  block  diagram  of  the  AN/MSQ-103  Receiver  Set,  Special  Purpose  (TEAMPACK 
Assembly)  showing  source/system  and  user/system  interfaces  and  the  (input) /output  data. 
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Figure  27.  Simplified  functional  block  diagram  of  the  Advanced  QUICK  LOOK  system  showing 
source/system  and  user/ system  interfaces. 


Figure  28.  Further  simplification  to  the  functional  block  diagram  of  the  Advanced  QUICK  LOOK  System 
showing  source/system  and  user/system  interfaces  and  the  (input) /output  data. 


and  the  data  analysis  subsystem,  with  capability  for  at  least  10,000  EOB 
records  and  10,000  combined  now  emitter  and  uncorrelated  LOB  records. 

As  with  the  TEAMPACK  Assembly,  the  receiving  antennas  of  the  airborne 
subsystem  denote  the  interface  between  the  system  and  sources  of  rf  emissions 
'  source/sy stem  interface!.  As  explained  earlier,  however,  more  convenient 
access  to  the  "source/system  interface"  may  be  realized  by  working  at  the 
interface  between  the  control  system  for  the  SLF  transmitters  and  the  SLF 
testing  control  program  'or  application  program,  shown  in  Figure  16).  Informa- 
shown  'at  the  interfaces),  in  Figure  28,  in  parentheses  denotes 
information  provided  by  the  user  to  the  system;  the  remaining  information  is 
by  tr.e  system  to  the  user,  as  described  earlier. 
r,":r. in.-  to  follow  the  structured  approach  to  describing  system 
m  u:  ar. :  the  development  of  measures  of  functional  performance,  user- 

functions  of  the  TEAMPACK  Assembly  and  the  Advanced  QUICK  LOOK  System 
•  „  id---ut  if  led.  The  generic  functions  identified  in  Table  2  for  electronic 
:  II  an  systems  arc  applicable.  These  functions  are: 

3igr..-«l  detection 


signal  characterization 

emitter  identification  and  location 

■'  -  ing  th  it  only  tr.e  Advanced  QUICK  LOOK  System  is  able  to  determine  emitter 
locations.  ;T,he  TEAMPACK  Assembly  determines  emitter  identifications  and  by 
comparing  these  characteristics  with  stored  data  that  include  known  locations 
e.f  unfriendly  emitters,  the  locations  of  intercepted  emitters  may  be  inferred.) 

Consistent  with  the  structured  approach  to  describing  system  performance 
and  as  already  noted,  tnere  are  specific  inputs  and  resultant  outputs  for  each 
fur, '0  ion.  A  generic  set  of  outcomes  is  discussed  in  Section  U  and  illustrated 
in  Figure  ,  Tr.e  possible  outcomes  normally  distinguished  are: 


Intended  Performance.  The  function  is  completed  within  a  specified 


maximum  performance  time  and  the  result 

.  im.it  intended. 

or 

outcome 

is 

within 

the 

'  r  " o ■' r e o t  Performance.  The  function 

.'p-?  ;  i  Tied  maximum  performance  time,  but 

i  s 

the 

completed 
result  or 

within 

outcome 

the 

in 

re  tne  limits  intended. 


vnrer : or 


tar/ 


unction  is  not  completed  within  a  specified 


o ^ c  f  ) rmanc?  t. i n ^ . 
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eri’  outcome,  specific  parameters  are  defined  that  describe 
the  TEAMPACK  Assembly  and  Advanced  QUICK  LOOK  System  relative  to 
unction  and  outcome. 

,  fir-st,  the  signal  detection  function.  Desired  or  intended 
curs  when  a  signal  is  present  and  is  detected  within  a  specified 
.  i'n  time.  Evidence,  to  a  user  (or  test  controller)  of  signal 
■m  ini'ial  intercept  alarm  such  as  that  provided  by  the 
i  y .  ioured  operating  frequency  also  is  provided  as  an 

.  ;  ,r  -f  *  no  video  detector  unit  of  the  TEAMPACK  Assembly  and 
.*  the  indicator-processor  unit.  (Measured  operating 
:  ■■  r.  -.r ;  *  *  r-r  must  be  within  acceptable  limits  of  the  actual 
:  •  t-.  t*  ••••'liter. )  The  relevant  parameter  is  detection  time 

:•  •• ,  f  'ossion  of  detection  trials)  which,  then,  is  a 

•  'tier,  efficiency  (or  speed) . 
i r.->  m cjrs  when  rf  noise  or  interference  is  mistaken  for 
in  ‘’it,  no  test  signal  is  present.  The  parameter  that 
.•  •  r^e-t  performance  is  false  detection  probab i  1  i ty ,  P(False 
parameter  can  also  be  thought  of  as  the  probability  of 
condition  that  no  signal  is  present  expressed  as  P(D|S). 
ise f a  1  measure  of  signal  detection  accuracy ,  is  calculated 
.1* >s  the  ratio  of  the  number  of  incorrect  (false) 
• *  1  n  .m b-r  of  detection  attempts. 

wi'.  l  indicated  when,  because  of  noise,  interference,  or 
ignal  is  not  detected  within  a  specified  maximum  detection 
rhst  •••  a^acter iocs  nonperformance  and  that  can  be  calculated 
r-ument  data  is  the  nondetection  probability ,  P(NonDetection) .  In 
•  •iso,  t-.is  parameter  is  the  probability  of  nondetection  given 

•  -m  >  r  i  rr:  I  is  present,  expressed  as  P(D|S).  This  parameter,  a 
!'  T/s'.’m  -"•tior  d°pen  Tab i  1  i ty ,  is  calculated  from  the  test 
•  ,t  ir  -f  *  •/  number  of  nondetentions  to  the  total  number  of 

..  :  ovet-'-m  mer f  rmanoe  parameters  for  the  signal 

•  i  -  •.  r-  •  i  '  ..v‘ rated  in  the  sample  space  diagram  shown  in 

*  .  ’in--  ••  'ndi*  ions,  the  sou""®  'system  inputs  for  signal 
■  •  a:.  '  -o'  signals  a  r.  d  a  "search  e  mtmand"  to  the  gnT.  The 


user/system  output  for  that  function,  as  discussed  above,  is  some  detection 
indication  for  the  intercepted  emitter,  for  example,  the  initial  intercept 
alarm  or  an  indication  of  measured  operating  frequency  as  provided  by  the 
TEAMPACK  Assembly.  Operating  frequency  also  is  provided  as  an  output  via  the 
information  display  and  control  unit  for  the  Advanced  QUICK  LOOK  System. 

An  illustration  of  the  signal  detection  function  with  inputs  and  output 
and  a  truth  table  that  relates  input  states  with  function  outcomes  is  shown  in 
Figure  30.  Vector  representations  of  the  input  states  use  "1"  for  the  ON 
condition  and  "0"  for  the  OFF  condition.  Similarly,  in  the  output,  "1" 

represents  DETECTION  INDICATED  and  "0"  represents  DETECTION  NOT  INDICATED. 
Input  state  (1,1),  then,  represents  a  trial  condition  in  which  the  test  signal 
is  ON  and  a  search  command  has  been  given  to  the  SUT.  An  associated  output 
con  lit  ion  (1)  represents  intended  performance  and  condition  (0)  represents 
: r.per fcrr.ar.ce .  Some  time  limit,  specified  by  the  user  (or  test  controller)  or 
o.ilt  into  the  system,  is  allowed  for  the  detection  function  to  be  completed. 
Wr.*»n  this  time  elapses  without  a  detection  indication,  nonperformance  is  the 
'  come.  input  state  (1,0)  is  meaningless,  since  it  represents  a  condition  in 
whirr  the  test  signal  is  ON  but  no  search  command  is  given — hence,  there  is  no 
trial.  Input  state  (0,1 )  represents  a  condition  in  which  the  test  signal  is 
when  a  search  command  is  given.  An  associated  output  condition  (0) 
rep  r?  s ‘-r.tr.  intended  performance  and  condition  (1)  represents  incorrect 


As  discussed  above,  the  parameter  that  quantifies  intended  performance  for 
e  ton  trial  is  detection  time,  the  elapsed  time  between  issuing  the  search 
command  and  receiving  a  detection  indication.  Mean  or  average  detection  time 
would  characterize  a  test.  A  succession  of  trials  could  be  quantified  by  the 
p  arameter  detection  rate.  Incorrect  performance  is  characterized  by  the 
parameter  f  a  lse  detection  probability,  P(False  Detection),  which  is  calculated 
from  test  results  as  the  ratio  of  the  number  of  false  detections  to  the  total 


of  detection  attempts.  Nonperformance  is  characterized  by  the  parameter 
v  o  tier,  prebab  i  1 1 1  y  ,  P  (NonDetec  t  ion) ,  which  is  calculated  from  test 


'...It-.  i.  the  ratio  of  the  number  of  nondetections  to  the  total  number  of 
■feet  ion  attempts.  (Of  course,  the  complement  of  the  sum  of  these 
'nubilities  is  the  probability  of  detection.) 

lo'-oir^  i  or  intended  performance  for  the  signal  characterization  function 
1  iz*'d  when,  for  a  detected  signal,  signal  characteristics  (carrier 
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(INPUTS)  (OUTPUT) 


Illustration  of  the  signal  detection  function  and  associated  input/output  truth  table 


frequency  and  determination  that  the  signal  is  either  CW  or  pulsed  and,  if 
pjlsed,  the  PW  and  PRF/PRI)  are  determined  within  a  specified  maximum  time  and 
the  measured  values  are  within  acceptable  limits  for  each  signal  characteris¬ 
tic  The  relevant  parameter  is  characterization  time  which,  then,  is  a  useful 
n-asure  of  signal  character ization  efficiency  (or  speed) .  Intended  performance 
:  it  also  occur,  logically,  when  the  SUT  is  given  a  command  to  characterize  a 
si '~iit  does  not  because  there  is  no  test  signal  input. 

Incorrect  signal  characterization  performance  occurs  when,  for  any  reason, 

v'  n;  ise,  interference,  multipath,  etc.,  the  measured  characteristics 
•  •  'ted  signal  are  not  within  acceptable  limits  for  the  test  signal  or 

is. .red  r.aracter  1st ics  are  indicated  though,  in  fact,  there  is  no  test  signal 
: r:  .  The  parameter  that  characterizes  this  incorrect  system  performance  is 

at-  i  1  i  ty  of  incorrect  signal  characterization,  P(  Incorrect  Character iza- 
a  useful  measure  of  signal  character  ization  accuracy,  this  parameter 
ul  .late:  from  the  test  results  as  the  ratio  of  number  of  incorrect  signal 
.  :  ’ter i:-  Jtior.s  to  the  total  number  of  signal  characterization  opportunities 

-.■’.•Is  f  ?r  which  a  signal  has  been  detected  and  the  SUT  has  been  "instructed" 

■  -•■'form  the  signal  characterization  function. 

•Nonperformance  for  the  signal  characterization  function  occurs  when,  for 
ison,  signal  characteristics  of  a  detected  signal  are  not  measured  within 
specified  maximum  performance  time.  The  parameter  that  characterizes 
n re r romance  for  this  function  is  the  probability  of  signal  noncharactei — 
i  :  : •  .  •  ■ . ,  P(fionSharacterization) .  This  parameter  is  calculated  from  the  test 
results  as  the  ratio  of  the  number  of  nonperformance  trials,  for  this  function, 
to  the  total  number  of  performance  opportunities  (trials  for  which  a  signal  has 

. or.  detected) .  The  parameter  is  a  useful  measure  of  dependability  for  the 

s :  '-*n.u  1  character  ization  function. 

"ur.  1 1  i on  outcomes  and  system  performance  parameters  for  the  signal 
ih-tracter ization  function  are  illustrated  in  the  sample  space  diagram  shown  in 

■■  cure  j  '  . 

Trough  tne  signal  characterization  function  is  a  little  more  complex  than 
tv*  detect:-  n  function,  the  source/system  inputs  still  are  known  test  signals 
:  i  "character izat ion  command"  to  the  SUT.  The  user/system  outputs  for  the 
function,  as  discussed  above,  are  the  signal  characteristics  of  carrier 
f^eouveey  -and  determination  that  the  signal  is  either  CW  or  pulsed  and,  if 
-  .  tne  PW  and  PRF/PRI.  For  the  TEAMPACK  Assembly,  these  data  are  provided 
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SIGNAL 

CHARACTERISTICS 
NOT  DETERMINED 


SIGNAL 

CHARACTERISTICS 

DETERMINED 

INCORRECTLY 

PROBABILITY  OF 
INCORRECT  SIGNAL 
CHARACTERIZATION 


INCORRECT 

PERFORMANCE 


PROBABILITY  OF 
SIGNAL 
NONCHARACTERIZATION 


NONPERFORMANCE’1 


*  Nonperformance  occurs  when  no  measurements  are  made 
of  signal  characteristics,  given  a  signal  has  been  detected. 


Figure  31.  Definitions  of  function  outcomes  and  system  performance 
parameters  for  the  TEAMPACK  Assembly  and  Advanced  QUICK 
LOOK  System  for  the  signal  characterization  function. 


76 


mmsmm m 


visual  outputs  by  the  pulse  train  separator  and  as  calibrated 
yj  the  indicator-processor  unit.  The  IPU  provides  visual  read-out  of 
or  the  data  nay  be  directed  to  the  teleprinter  for  hard  copy.  The 
lector  unit  also  provides  audio  output  as  a  monitor  of  the  PRF. 
ted  signal  characteristics  are  provided  by  the  Advanced  QUICK  LOOK 


put  to  the  user  via  the  data  analysis  subsystem's  information 


intro!  unit.  Information,  then,  may  be  directed  to  the  video 
printer  for  hard  copy.  (Information  may  also  be  directed  to 


-*o-reel,  or  disk  storage  devices.) 


ion  of  the  signal  characterization  function  with  inputs  and 


truth  table  that  relates  input  states  with  output  states  is 


The  same  vector  representations  of  the  input  states  are 


or  t r.---  detection  function.  In  the  output,  "1"  indicates  the 


be  provided  and  "0"  indicates  the  signal  character¬ 


ing.  doing  the  vector  order  as  (FREQ, CW, PW, PRF/PRI ) ,  the 
•ins  character istics  for  a  pulsed  signal  have  been  measured, 
'1,1, 0,0)  means  character ist ics  of  a  CW  signal  have  been 
I v ,  many  of  the  logically  possible  output  state  vectors  have 
g,  such  as  the  vector  (0,0, 1,1)  which  would  indicate  that  PW 
ci3. iron  v’ut  there  was  no  measurement  of  frequency, 
above,  tne  parameter  that  quantifies  intended  performance, 
imacterization  function,  for  each  trial,  is  character ization 
d  time  between  issuing  the  characterization  command  and 
ation  of  signal  character  ization.  Mean  or  average 


vouid  character ize  a  test.  Referring  to  the  information 


32  of  possible  input  and  output  information,  intended 


when  the  output  information  denoted  by  vectors  (1,0, 1,1) 
^  or  (I,1, 3,0)  (for  a  CW  signal)  is  correct  for  the  input 


i,’-1,  or  the  input  state  (0,1)  results  in  an  output 


’ect  performance  is  characterized  by  the  parameter 


:t  signa 1  character ization,  P( Incorrect  Character iza- 


:  from  tost  results  as  the  ratio  of  the  number  of 


■  izntions  to  the  total  number  of  signal  character iza- 


'tef erring  to  the  information  shown  in  Figure  3?.  incorrect 


er  i  /. at  ior 


occurs  when  tne  output  information  denoted  by  vectors 


■  i  •  o  0  > 

-  1  i  9  9  ' 


incorrect  given  the  input  conditions  denoted  by 
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MV  j 


(INPUTS) 


(OUTPUTS) 
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TEST 

1-ON 

SIGNAL(S) 

0  —  OFF 

SIGNAL 

CHARACTERIZATION 

CHARACTERIZATION 

1-ON 

FUNCTION 

COMMAND 

O-OFF 

1  —  INDICATED 


•FREQUENCY 

•SIGNAL  TYPE 

-  CVJ 

-  PULSED 

•  PW 

•  PRF/PRI 


POSSIBLE  VECTORS  OF  INPUT  INFORMATION 


(TS,  CC) 


(Event  Order) 


(O.K.) 

(No  Trial) 

(O.K.) 

(Nothing) 


POSSIBLE  VECTORS  OF  OUTPUT  INFORMATION 


(Fr«q.  CW.  PW.  PRF/PRI)  (Event  Order) 


(0, 0.0,0)  (Correct  or  nonperformance) 


(0,0, 0,0) 
(0,0.0,  1 )' 
(0,0. 1.0) 
(0,0, 1,1) 
(0,1, 0,0) 
(0,1,0,11 
(0,1,  1.0) 
(0,1, 1, 1 ) 
(1.0, 0,0) 
(1,0,0, 1) 
(1.0.1, 0) 
(1,0,1,  1) 
(1. 1.0,0) 


(Nonsense) 


<1,1.0, in 
(1.1, 1,0)^ 
o.i.i.nj 


(Incorrect  or  nonperformance) 
(Nonsense) 

(Incorrect  or  nonperformance) 
(Correct  or  incorrect-CW) 
(Correct  or  incorrect- pulsed) 


(Nonsense) 


INPUT/OUTPUT  TRUTH  TABLE  FOR  SIGNAL  CHARACTERIZATION  FUNCTION 


FUNCTION 
"''"^^^OUTCOMES 
INPUT^^^ 
STATE  VECTORS-"-^. 


INTENDED 

(EFFICIENCY) 


INCORRECT 

(ACCURACY) 


NONPERFORMANCE 

(DEPENDABILITY) 


(1,0,1,  1  Hpulsed) 
(1, 1,0,0)<CW) 


(1,0,1, 1) 
(1,1, 0,0) 


(0, 0,0,0),  (1.0.0.0) 
(1,0, 1,0) 


(0,0, 0,0) 


(1,0, 0,0) 
(1,0, 1,0) 
(1,0,1, 1) 
(1,1, 0,0) 


Figure  32.  Illustration  of  the  signal  characterization  function  and 
associated  input/output  truth  table. 
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'  of  the  input  vectors  (1,1)  or  (0,1).  In  addition,  the  output  vectors 
',j'  md  (1,0, 1,0)  denote  incorrect  performance  for  the  (0,1)  input  vector 
Nonperformance  is  characterized  by  the  parameter  probability  of  signal 
^rasterization,  P(Noncharacterization) ,  which  is  calculated  from  test 
as  the  ratio  of  the  number  of  signal  noncharacterizations  to  the  total 
of  signal  characterization  opportunities.  Again,  referring  to 
,  signal  noncharacterization  occurs  when  output  information  is  denoted 
.<  t  .rs  (0,0,0,01,  (1  ,0,0,0),  or  (1,0, 1,0)  given  the  input  condition 

:y  t-.e  vector  (1,1). 

•'■mitter  identification  and  location  function  is  still  more  complex, 
intended  performance  for  this  function  is  possible  only  when  desired 
has  occurred  for  the  detection  and  signal  characterization  func- 
. ‘sired  cr  intended  performance  for  this  function,  then,  requires  the 
■  •-  .ai'-ulation  of  at  least  one  true  bearing  for  an  intercepted 


t  7- AM PACK  Assembly,  the  function  is  completed  when  the  system 
frequency,  signal  characteristics,  and  line  of  bearing  data 
.red  data  for  known  systems.  Intended  performance  occurs  when  the 
me  a  a  ,.r  ■  mer.t  /c  a  leu  1  a  t  i  on  and  data  comparison  are  completed  within  a 
i  maximum  t  ime  ml  the  measured/calculated  line  of  bearing  is  within 
I: -it s  for  the  true  bearing. 

t.ue  Advanced  QUICK  LOOK  System,  the  process  can  extend  to  the 
il  measurement  and  calculation  of  several  true  bearings  to  calculate 
location  as  the  intersection  of  these  lines  of  bearing,  within  the 
if  acceptable  EED.  As  has  been  noted  earlier,  however,  "successful" 
rme  may  produce  data  in  three  different  categories,  namely 

frequency,  signal  character i st i cs ,  and  location  for  known 

emitters 

frequency,  signal  characteristics,  and  location  for  new  emitters 

frequency ,  signal  characteristics,  and  true  bearing  for  known  or 

••••.;  emitters. 


'  *  dp  if]  3  o 


is  realized  when  the  "expected  outcome"  (identification 


a  'f  a  known  emitter,  identification  and  location  of  a  new  emitter, 
leaf  ion  and  true  bearing  for  either  a  known  or  new  emitter)  is 
mi  thin  a  specified  maximum  Derformanoe  time  and  the  measured/ 


calculated  true  bearing  or  location  values  are  within  acceptable  tolerance 
limits. 

The  time  required  to  measure/calculate  one  or  more  true  bearings  and/or 
the  time  required  for  the  system  to  compare  the  measured  data  with  stored  data 
'this  time  would  include  the  time  to  measure/calculate  a  true  bearings),  that 
is,  the  emitter  identification  and  location  (EIL)  time  is  a  parameter  that  can 
be  meas ured/caleulated  as  a  useful  measure  of  the  efficiency  (or  speed)  of  the 
system,  in  performing  the  EIL  function. 

Incorrect  emitter  identif ication  and  location  performance  occurs  when,  for 
any  reason  such  as  rf  noise.  Interference,  multipath,  etc.,  the  individual 
m-‘ as urener.t  calculation  of  true  bearing  for  an  intercepted  emitter  is  not 
witnir.  acceptable  tolerance  limits  for  the  true  bearing  of  the  emitter,  though 
Section  and  signal  character izat ion  functions  have  seemed  to  be  performed 
.  ■  vv  sful ly .  In  situations  where  several  true  bearing  measurements  are  used 
t-  •-■st  iMish  location  as  the  intersection  of  these  lines  of  bearing,  incorrect 
••rformance  occurs  when  the  calculated  EEP  exceeds  some  maximum  limit  (speci¬ 
fied  :y  the  user  or  test  controller)  or  the  measured  lines  of  bearing  do  not 
■•v  intersect.  The  parameter  that  character izes  this  incorrect  performance  is 
the  p~  scab i 1 i ty  of  incorrect  EIL  or  P(Incorrect  EIL).  As  a  useful  measure  of 
emitter  identification  and  location  accuracy,  this  parameter  is  calculated  from 
t h  *  test  results  as  the  ratio  of  the  number  of  incorrect  measurements /calcula¬ 
te  sns  of  Eli.  to  the  total  number  of  measurement  opportunities  during  the  test 
'  trials  for  which  the  signal  has  been  detected  and  characterized  and  the  SUT 
nas  been  "instructed"  to  perform  the  emitter  identification  and  location 
f  i ;  t.  i  on ) 

Nonperformance  for  the  emitter  identification  and  location  function  occurs 
w-:*-n,  f ;r  any  reason,  emitter  "identification  and  location"  are  not  determined 
witr.ir.  a  specified  maximum  time  when  seemingly  successful  detection  and  signal 
char  no  te*'  i  z  at  ion  have  occurred.  Depending  on  the  SUT,  this  function  may  be 
r 1  a- i v<‘\ y  simple  or  relatively  complex.  For  example,  the  function  may  entail 
-oasurement  of  a  line  of  bearing  for  a  detected  and  characterized 
1 1  yr.  ,  1  «r.d  the  comparison  of  these  measured  data  with  stored  data  for  known 
systems  such  as  is  done  by  the  TEAMPACK  Assembly.  Or  the  function  may  require 
the  measurement  of  at  least  two  lines  of  bearing,  the  calculation  of  emitter 
location  using  triangulation  techniques,  and  the  comparison  of  the  measured 
1  :ta  witt  stored  data  for  known  systems  to  determine  if  the  measured  data 


new 


nown  or  new  emitter  such  as  is  done  by  the  Advanced  Q'JICK  LOOK 
parameter  that  characterizes  nonperf ormance  for  the  EIL  function 
> i I i t y  o£  nor.- SI L  or  P(Non-EIL).  As  a  useful  measure  of  system 
in  performing  the  emitter  identification  and  location  function, 
r  is  calculated  from  the  test  results  as  the  ratio  of  the  number 
fcrnar.ce  trials  to  the  total  number  of  performance  opportunities 
aieh  the  detection  and  signal  characterization  functions  have  been 
.cessfuliy  an:  the  SUT  has  been  "instructed"  to  perform  the  EIL 


outcomes  -r.d  system  performance  parameters  for  tne  emitter 
■.  and  loC"tL~o  function  are  illustrated  in  the  sample  space 

Figure  7_-. 

'system  inputs  for  the  emitter  identification  and  location 
Known  test  signal,  measured  signal  characteristics  for  the 
from  too  signal  characterization  function),  and  an  "EIL  command" 
The  user/ system  outputs  for  the  function,  as  discussed  above,  are 
•io'-.al  ch  '.rioter istics  and  a  1  i ne-cf-bear ing  measurement  or  a 
tins  an  1  associated  EEP,  depending  on  the  SUT.  From  a  func- 
Lnt,  th<  To AMPATK  Assembly  actually  provides  only  emitter 
whereas  tro-  Advanced  QUITE  LOCK  System  provides  both  emitter 
,n  ■  ion ,  using  data  f-om  the  inertial  navigation  systems  on 

It,  the-vf  .--"e,  is  useful  to  discuss  these  systems  separately  for 


•.'K  Assembly  compares  the  measured  frequency,  signal 
an:  true  t taring  with  the  stored  parameters  and  locations  of 
establish  emitter  identification.  The  user/system  outputs  for 
en,  are  indicated  bearing  and  an  emitter  number  that  identifies 
v'-.  ieh  the  measured  character  ist  ics  and  bearing  compare,  within 
f  tolerance  for  each  parameter  of  each  "identified  system"  or 
a*  no  emitter  for  which  stored  characteristics  compare 

lit:.  it  put  to  the  user  of  these  data  is  provided  as 

ion  or.  the  7DU  and  as  calibrated  information  on  the  IPU. 

;  r.oy ,  'oration,  and  identification  data  provided  by  the  IPU 
t y  tr.e  teleprinter,  either  as  single  sets  or  as  the  total 
‘■d  ir.  tn-  computer's  mission  memory. 


DESIRED/ INTENDED 
PERFORMANCE 


i 


EMITTER  IDENTIFIED 
and 

LOCATED  CORRECTLY 


EMITTER  IDENTIFICATION  and  LOCATION  TIME 


EMITTER 

IDENTIFIED/LOCATED 

INCORRECTLY 


PROBABILITY  OF 
INCORRECT  EMITTER 
IDENTIFICATION 
and  LOCATION 


INCORRECT 

PERFORMANCE 


EMITTER 
NOT  IDENTIFIED 
or  LOCATED 

PROBABILITY  OF 
EMITTER  IDENTIFICATION 
and  LOCATION 
NONPERFORMANCE 


NONPERFORMANCE 


*  Nonperformance  occurs  when  no  measurements 
are  made  of  true  bearing,  measured  signal 
characteristics  do  not  match  stored  characteristics 
of  known  systems,  or  emitter  location  is  not  determined 
by  triangulation. 


Figure  33.  Definitions  of  function  outcomes  and  system  performance 
parameters  for  the  TEAMPACK  Assembly  and  Advanced  QUICK 
LOOK  System  for  the  emitter  identification  and  location 
function. 
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:'K  System  also  compares  measured  frequency  and  signal 
::  information  for  known  systems  to  perform  emitter 
ion,  triangulation  calculations  are  performed  using 
measurements  to  establish  1 ine-of-bear ing  intersee- 
.ermine  emitter  location,  within  acceptable  limits  of 
between  the  measured  frequency  and  signal  character¬ 
ed  the  elliptical  error  probability  resulting  from 
■  of  bearing.  Data  thus  measured  and  calculated  are 
' known  systems  (the  so-called  HOB  data),  (2)  data 
-determined  locations,  or  (3)  data  for  uncorrelated 
visured  frequency,  signal  characteristics,  and  true 
■■late  with  stored  information  for  known  emitters  nor 
■•r.ersect  to  establish  new  emitter  locations  within 
These  data  are  provided  as  visual  read-out  to  the 
iunsy stem)  information  display  and  control  units,  or 
the  video  copier  or  line  pointer  for  hard  copy.  As 
’.so  may  be  stored  on  any  of  several  magnetic  storage 
analyzed  by  the  ELINT  operator /analysts  are  relayed 
nation  systems  that  are  not  a  part  of  the  Advanced 

the  emitter  identification  and  location  function, 
to  the  function,  vector  representations  of  these 
froth  table  for  evaluating  the  function  outcomes,  is 
output  vectors  show  ( EMTR  NO, FREQ,  CW,  PW,  PRF/PRI  , 
"  to  denote  information  is  provided  or  missing,  as 
?  logically  are  64  possible  output  state  vectors; 

-  v-  states  represent  meaningful  trial  results.  These 
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for  the  (1,1)  input  state, 
o  measure  data  for  a  pulsed  or 


(INPUTS)  (OUTPUTS) 


r 

EIL  PARAMETERS 

TEST  1— ON  , 

EMITTER 
IDENTIFICATION 
and  LOCATION 
(EIL) 

NEW  EMITTER  OATA 

KNOWN  EMITTER  OATA 

SIGNAL(S)  0  —  OFF 

SIGNAL 

•  FREQUENCY 

•  SIGNAL  TYPE 

•EMITTER  NO. 
•FREQUENCY 

CHARACTERISTICS 

•  fu; 

-PULSED 

•  SIGNAL  TYPE 
-CW 

EIL  1  —ON  . 

COMMAND  o  —  O  FF 

FUNCTION 

•  PW 

• PRF/PRI 
•LOB/EEP 

-PULSED 

•  PW 

• PRF/PRI 
•  LOB/EEP 

POSSIBLE  VECTORS  OF  INPUT  INFORMATION  POSSIBLE  VECTORS  OF  OUTPUT  INFORMATION 


(TS,  EC)  (Event  Order) 


.1,  1) 

(O.K.) 

(1,0) 

(No  Trial) 

(0.1) 

(O.K.) 

(0,0) 

(Nothing) 

(EMTR  NO..  FREQ,  CW.  PW.  PRF/PRI.  LOB/EEP)  (Event  Order) 
64  viclon  logically  an  ponlbli. 

Only  liven  victor*  dinoti  meaningful  trial  output*. 


i.Yo.oAon 

>.1.0,1.  I,0)J 


(0,0,0.0,0,01 
<0, 

(0, 

<0,1,0, 1,1,1) 
(0,1,  t,  0,0,1) 
(1. 1.0.1, 1,1) 
(1.1,1, 0,0,1) 


(Cornet  or  nonpirformanci) 

(Income!  or  nonpirformanci) 

(Cornet,  Incornct  or  nonpirformanci) 

(Cornel  or  Incornct  performance) 


INPUT/OUTPUT  TRUTH  TABLE  FOR  EVALUATING  FUNCTION OUTCOMES 


FUNCTION 

^■"-^OUTCOMES 

INPUT 

STATE  VECTORS 

INTENOEO 

(EFFICIENCY) 

INCORRECT 

(ACCURACY) 

NONPERFORMANCE 

(DEPENDABILITY) 

(1.  1) 

(EIL  data  found  In 
•  land  data) 

(1, 1,0,1, 1, 1 )  (pultid) 
(1.1,  1,0,0, 1  )(CW) 

(1, 1,0,1,  1,  1  )  (pul  ted ) 
(1,1, 1,0, 0,1  )(CW) 

(O.O.O.O.O.O) 

(0.1. 0,0, 0.0) 

(0.1. 0.1,  1,0) 

<0.1.0, 1,1,1) 
(0,1,1,0,0,11 
(1. 1,0.1.  1, 1)(CW) 

( 1.1.1. 0,0,1  Hpulnd) 

(1,1) 

(EIL  data  not  found 

In  tlond  data) 

(0,1,0, 1, 1,1)  (pulsed) 
(0,1,1,  0,1, 1HCW) 

(01011  1  Kpultid) 

(0,1,1, 0,0, 1MCW) 

(O.O.O.O.O.O) 

(0,1,0, 0,0,0) 

(0.1, 0,1,  1,0) 
(0,I,0,1,I,I)(CW) 

(0,1,  l.o.o.l ) (pultid) 
11,1.0,1.1,1) 

(1.1. l.o.o.l) 

(0.1) 

(0,0, 0,0, 0,0) 

(0,1, 0,0, 0,0) 
10,1,0,1,1,0) 
10,1,0,1,1,1) 

(0,1, 1,0,0, 1) 

(1,1, 0,1, 1,1) 

(t, 1,1, 0,0,1) 

Figure  34.  Illustration  of  the  emitter  identification  and  location  (EIL) 
function  and  associated  output  data  truth  table. 
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-r.  inter  tnat  is  in  the  stored  data  file,  (2)  correct  or 
’rect  performance  in  measuring  data  for  a  pulsed  emitter  that 
;-t  in  the  stored  data  file,  or  (3)  nonperformance  in  attempt- 
to  measure  data  for  a  CW  emitter  that  is  not  in  the  stored 
file  and  incorrect  performance  for  tne  (0,1)  input  state 


,’,0,0,1)  representing,  for  the  (1,1)  input  state, 

.  r. performance  in  attempting  to  measure  data  for  a  pulsed  or 
•m  it  ter  that  is  in  the  stored  data  file,  (2)  correct  or 
“rent  performance  in  measuring  data  for  a  CW  emitter  that  is 
in  the  stored  data  file,  or  (3)  nonperformance  in  attempting 
lata  for  a  pulsed  emitter  that  is  not  in  the  stored 
ar.d  in  -orrect  performance  for  the  (0,1)  input  state 


presenting,  for  the  (I,1)  input  state, 
•e-t  or  incorrect  performance  in  measuring  data  for  a 
ter  --.at  is  in  the  stored  data  file,  (2)  nonperform- 
.tt-’mp- 1  rg  to  measure  data  for  a  CW  emitter  that  is  in 
'  da*  :  file,  or  (3)  nonper formance  in  attempting  to 
1  s*  i  f  •’  -  pulsed  or  CW  emitter  that  is  not  in  the  stored 
•  :  inc  '-’rent  performance  for  the  (0,1)  input  state 


,  ? ,  '  representing,  for  the  (1,1)  input  state, 
f incorrect  performance  in  measuring  data  for  a  CW 
■tat  in  in  the  stored  data  file,  (2)  nonperformance  in 
to  measure  data  for  a  pulsed  emitter  that  is  in  the 
iti  file,  or  (3)  nonperformance  in  attempting  to  measure 
a  pulsed  or  CW  emitter  that  is  not  in  the  stored  data 
inco-rect  performance  for  the  (0,1)  input  state. 


d,  the  parameter  that  quantifies  intended  performance 


Lon,  for 


■Mch  trial,  is  STL  time,  the  elapsed  time  between 


dommm.i  and  receiving  indication  of  emitter  identification  and 


ar  average  El!,  time  would  characterize  a  test.  Referring  to 


;  fr-’mat  ion  discussed  above,  intended  performance  occurs  when  the 
uhlan  den  tt-'-d  by  vectors  (  1 , 1  , 0 , 1  , 1  , 1  )  or  ( 1  , 1  , 1  ,0 ,0 , 1  )  is  correct 


.  th  stored  data  or  when  the  output  information  denoted  by 


' " , 1 , 1 ,0 ,0 , 1  )  is  correct.  (There  are  no  stored  data 


-.ensured  data.) 


is  characterized  by  the  parameter  probabi lity  of 


E  L  L ) ,  wh i 


calculated  from  the  test  results  as 


tter  identifications  and  locations  to 


■■por tun i ties.  The  same  output  vector  states  apply  as 


The  difference  is  that  the  information  is  incorrect 


oes  not  correlate  with  stored  data  when  it  should,  or 


■or*'!  d  it  a  wh'fi  it  should  not. 


Nonperformance  is  characterized  by  the  parameter  probability  of  non-EIL, 
?  Nor:- SI L' ,  which  is  calculated  from  the  test  results  as  the  ratio  of  the 
r.umhe"  of  non-EIL  trials  to  the  total  number  of  EIL  opportunities.  The  output 
vector  states  that  always  denote  nonper f ormance  are  ( 0 , 1  , 0 , 0 , 0 , 0  )  and 
1  *  1  .9  )  :  vector  states  (0 , 0 , 0 , 0 , 0 ,0  > ,  (0 , 1  ,0 , 1  , 1  , 1  ) ,  ( 0 , 1  , 1  , 0 ,0 , 1  ) , 

’ , '  ,  and  ( 1 , 1  , 1  , 3 ,0 , 1 )  sometimes  also  denote  nonperformance,  as  shown 

in  F i oure  •  . 

Prir.v'y  v  lrameters  associated  with  the  system  functions  signal  detection, 
s ‘or;-.:  ira  .'"e- izat  ion,  and  emitter  identification  and  location  have  been 

i  rah  .fi'r:  .nd  discussed.  These  parameters  describe  system  performance  under 
ap-rr  iti ng  conditions.  Each  performance  trial  will  produce  an  outcome 
at  least  one  function,  as  we  have  discussed.  We  recognize,  however,  that 
■  •s.cful  emitter  identification  and  location  is  possible  only  when  the  signal 
:  r;  d  signal  characterization  functions  have  been  completed 
;  ••••.:  -  fully,  and  successful  system  performance  is  realized  when  this  func- 
•  :  : . a  1  process  is  successfully  completed  for  a  specified  fraction  of  the 
••rfc-mar:  trials.  That  is,  "successful"  system  performance  is  manifest,  from 

.  user’s  perspective,  when  the  intended  outcomes  for  all  functions  are  realized 
•’  m  :  specified  fraction  of  the  performance  trials  (a  threshold  or  minimum 

1:.  «:•  r-izaticr:  of  the  long-term  system  performance  through  aggregation 
erf  rmtr.ee  results  over  successive  performance  periods  is  accomplished 
u.-:  ;«e  f  the  availability  function,  as  illustrated  in  Figure  21,  and 

.ns cc i  ite  :  secondary  parameters,  defined  in  Figure  22.  Successive  performance 
:  ••riois  -f  successful  -erforrnance  constitute  an  available  state;  similarly, 
successive  periods  of  unsuccessful  performance  constitute  an  unavailable  state. 
7  .  a  t  ion  a  1  states  are  related  mathematically  by  system  failure  rate,  >. , 

,  ;  j ;  r  3te,  y  or  their  reciprocals,  MTBF  and  MTTR,  as  discussed  in 

‘1  F?s  for  tr.°  TEAMPACK  Assembly  and  the  Advanced  QUICK  LOOK  Fys*- 
i  •  i;.  -  :  rim  -ry  and  secondary  parameters,  are  summarized  in  Table  A. 
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Table  U.  Measures  of  Functional  Performance  (MOFPs)  for  the  AN/MSQ-103 
Receiver  Set  (TEAMPACK  Assembly)  and  the  Advanced  QUICK  LOOK 
System 

FUNCTION  MOFP 


Signal 
Detect  ion 


gnu  i 


■  i  c  at  ion 


ient 1 f ioat ion 
ion  ’EIL) 


’*ibi  1  itv 


1 1 


Detection  Time  (or  Rate). 

Probability  of  False  Detection. 

Probability  of  Nondetection. 

Characterization  Time. 

Probability  of  Incorrect  Characterization.7 

Probability  of  Noncharacterization. 8 

Emitter  Identification  and  Location  (EIL)  Time. 

Probability  of  Incorrect  EIL.9 

Probability  of  Non-EIL.10 

Availability;  (A)  »  y  /  (y  +  X). 

(Time  that  the  System  is  in  an  Available  State.) 

Unavailability;  (U)  ■*  X  /  (y  +  X). 
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L* 


ijter ization  Includes  frequency  (from  the  detection  function)  or, 
i:  ••••:.;  I  red,  determination  of  frequency,  identification  of  the  signal  as  CW  or 
r  j.  ;  -:,  irvi,  if  pulsed,  determination  of  PW  and  PRF/PRI. 

’  lr. '  crrect  signal  character ization  occurs  when  measured  values  for  signal 
p i meter s  frequency,  PW,  and  PRF/PRI)  differ  from  known  values  by  more  than 
■  i  -.able  tolerances  or  measured  values  are  indicated  when,  in  fact,  no  input 
signal  with  those  characteristics  was  present. 

"  •  ~ ' 1 1  /  speaking,  incomplete  signal  characterization  occurs  when  measured 
for-  some  signal  parameters  are  missing.  This  condition  is  treated 

Logically  as  noncharacterization. 

-i-' t  ML  occurs  when  the  measured  value  for  a  LOB  differs  from  the  known 
.  M.  /  •  ~  than  an  acceptable  tolerance,  measured  values  for  several  LOB's 
.  •  ■  ions  that  yield  an  EEP  that  exceeds  an  acceptable  tolerance 

•  -  *  .  •  me-ts ire  1  vi’.ues  produce  an  incorrect  identification  when  compared 

•  ~i  *  ‘  ••r' s  '.'.own  characteristics,  or  a  new  emitter  is  identified  as  a 


l-iog,  incomplete  emitter  identification  and  location  occurs  when 
■<r  LLP  missing.  This  condition  is  treated  logically  as 


W  refill  »»■»«>, 


6.  PERFORMANCE  MEASUREMENT  APPROACH 

Communications-electronics  (C-E)  systems  that  will  be  tested  using  the 
Stress  Loading  Facility  (SLF)  will  be  very  complex  and  usually  will  incorporate 
computer  control.  The  structured  approach  taken  in  this  study  to  defining 
performance  parameters  for  these  systems  yields  parameters  that  are  character¬ 
ised  by  events  that  must  be  either  timed  or  counted.  Both  of  these  processes 
are  done  most  conveniently  by  using  a  computer  to  record  the  event  outcomes  and 
tabulate  or  calculate  system  performance  using  the  measures  of  functional 
performance  developed  in  Section  5.  In  that  section,  three  principal 
functions — detection,  signal  characterization,  and  emitter  identification  and 
loeation--are  defined  for  electronic  surveillance  systems.  Then,  11 
parameters — 9  defined  as  primary  and  2  defined  as  secondary — are  developed  to 
describe  functional  performance  of  the  system  from  the  perspective  of  the 
user's).  These  performance  parameters  (termed  measures  of  functional 
performance)  are  shown  in  Table  4).  The  primary  parameters  describe  system 
performance  with  respect  to  three  general  performance  criteria,  namely 
•efficiency  (or  speed),  accuracy,  and  dependability.  The  secondary  parameters 
Je scribe  long-term  or  aggregate  performance  in  terms  of  the  operational  state 
of  the  system  as  either  available  or  unavailable. 

The  process  for  testing  systems,  using  these  parameters  that  are  system 
independent,  is  illustrated  in  Figure  35.  Inputs  to  the  testing  process 
;\:rsist  of  testing  objectives,  defined  by  the  type  of  tests  being  performed, 

!  trial  outcomes  observed  at  the  user/system  interface(  s) .  Results  of 
testing  arc  the  estimated  (mean)  values  of  timed  events  and  the  calculated 
results  of  ratios  of  counted  events,  i.e.,  estimated  probability  of  detection 
'or  nondetection),  estimated  probability  of  incorrect  measurement  of  signal 
characteristics,  etc.  The  testing  process  is  accomplished  in  four  phases  which 
-e-  have  defined  as  test  design,  data  collection,  data  reduction,  and  data 
analysis. 

Test  design,  discussed  in  Section  6.1,  applies  general  test  objectives  in 
tnri  development  of  a  detailed  test  plan  that  defines  the  test  conditions  and 
v.f  i  f  i  o  system  performance  information  that  is  to  be  collected.  Data 
o'.:  lection,  discussed  in  Section  6.2,  describes  the  test  signals  that  are 
i  :uced  at  the  source/system  interface(s)  and  the  corresponding  trial 
•  ityjtc  'events)  that  are  monitored  at  the  user/system  inter face( s) .  Data 
-•  : :n,  discussed  in  Section  6.3,  is  the  merging  and  processing  of  the 


Generalized  illustration  of  the  system  performance  measurement  process  using  system-independent 
measures  of  functional  performance. 


collected  data,  perhaps  from  several  user/system  interfaces,  to  produce  the 
performance  results,  such  as  estimates  of  mean  time  (for  an  outcome)  and  the 
various  probabilities.  Data  analysis,  discussed  in  Section  6. *4,  is  a  process 
of  statistical  examination  of  the  reduced  data  to  determine  the  precision  of 
estimated  parameter  values  and  other  associated  conclusions. 

The  measures  of  functional  performance  and  the  SLF  test  methods  developed 
in  this  report  focus  on  performance  assessment  from  the  user’s  perspective. 
These  processes  are  quite  general  and  independent  of  many  implementation 
details.  In  addition,  there  are  other  important  tests  of  the  systems  that  may 
have  to  be  performed  using  other  testing  facilities.  An  example  is  the 
measurement  of  many  engineering-oriented  parameters  using  the  Instrumented  Work 
Shop.  We  identify  these  test  requirements  in  Section  7. 


6.1  Test  Design 

This  section  defines  general  procedures  for  designing  SLF  tests,  on  Army 
systems,  that  will  provide  estimates  for  the  parameters  used  as  measures  of 
rtional  performance  and  the  associated  testing  precision  and  variability 
Misties.  The  test  design  serves  as  a  guide  to  the  data  collection  process 
to  the  subsequent  data  reduction  and  analysis  processes  from  which 
'ormance  parameters  are  estimated  and  associated  precision  and  variability 
-.is tics  are  determined.  A  good  test  design  will: 


establish  well-defined  connections  between  the  test  results  and 
conclusions  and  decisions  that  will  be  made  based  on  the  test 


»  -  c1  •  I  ^  c; 


seek  to  avoid  bias  in  measured  values  (measurement  accuracy) 


guide  in  obtaining  the  desired  accuracy  (precision  in  measured 
values)  in  test  results 


assure  efficient  use  of  test  resources  (e.g.,  time  and  money). 


Tnr'  aooiication  of  statistical  methods  that  are  central  to  this  discussion 


r.f-st  design  requires  definitions  of  some  specific  testing  (or  measurement) 
me.  A  trial  is  an  individual  attempt  to  perform  the  sequence  of  the 


f'u.nc  t  ions , 


g.  ,  detection,  signal  character ization,  and  emitter 


if  ication  and  location  for  electronic  surveillance  systems.  A  population 


sot  of  all  trials  of  Interest  in  a  particular  test.  A  sample  is  a  subset 


of  the  population  actually  measured  during  a  test.  The  relationship  between 
these  terms  is  illustrated  in  Figure  36. 

A  factor  is  a  variable  that  describes  system,  application,  or  testing 
conditions  that  are  expected  to  influence  observed  performance  (measured 
values).  Levels  are  the  defined  states  or  values  for  a  factor  during  a  test. 
A  f actor  combination  is  a  set  of  specified  levels  for  each  factor  of  interest. 
Finally,  a  test  is  a  process  of  data  measurement  that  is  continuous  in  time  and 
involves  only  one  factor  combination.  Thus,  all  conditions  existing  during  a 
test  are  defined  by  a  specific  factor  combination. 

To  illustrate  the  meaning  of  these  terms,  consider  testing  of  the  TEAMPACK 
Assembly.  The  Functional  Specification  (Bunker  Ramo  Corporation,  1979) 
esi  1  ishes  engineer ing-or iented  performance  parameters  for  frequency  sweep 
time,  signal  process  time,  parameter  data  print-out  time,  the  separation  of 
i  f  feront ,  simultaneous  signals,  etc.  These  specifications,  then,  may  be  used 
to  define  a  test,  the  factor  combination  for  the  test,  each  trial  of  the  test, 
ir.d  the  test  sample  or  population  used  for  determination  of  system  performance. 
For  ample,  several  signals  may  be  simultaneously  irradiating  the  system  under 
test.  Each  trial  would  be  a  single  attempt  to  detect  a  signal  (including 
is, cement  of  the  carrier  frequency),  measure  characteristics  (pulse  width  and 
puls*-*  repetition  frequency)  of  the  signal,  measure  a  direction  of  arrival  for 


higr.u. , 


and  compare  these  data  with  stored  characteristics  of  known 


-'titters  ("identify"  the  emitter).  The  combined  frequency  sweep  time,  signal 
process  time,  data  print-out  time,  etc.,  will  determine  the  number  of  trials 
that  are  possible  in  a  defined  interval  of  time.  All  trials  (the  total  number 
-'f  opportunities  for  making  these  measurements)  comprise  the  population  of 
interest,  and  each  subset  of  trials  relating  to  attempts  to  detect,  character- 
i te,  identify,  and  locate  each  irradiating  system  (emitter)  comprise  the 


of  t.ne  population. 


Of  course,  signal  levels  for  all  signals  must 


n  constant  for  each  factor  combination. 


Th-.-  test  design  process  is  understood  most  easily  by  defining  and 
i  i-  ,-  o'  v-.ee :1  steps.  The  first  step  is  to  define  the  test  objectives. 

,  •  t;v  s  often  are  determined  most  effectively  by  identifying  the 


'■:)  steps.  The  first  step  is  to  define  the  test  objectives. 

often  are  determined  most  effectively  by  identifying  the 
v.Li  be  supported  by  the  test  results.  Examples  of  decisions 
on  Vn>-  test  (system  performance)  results  are  (1)  buy/do  not  buy 
stem  cost  effectiveness),  (2)  product  improvement  objectives 


*  v 

•  l 


.  >  ,  4  rn  * i 


8WWW 


have/have  not  been  achieved,  or  (3)  measured  performance  meets/does  not  meet 
expected  performance. 

There  are  three  general  types  of  tests  that  may  be  conducted  by  following 
the  methodology  outlined  in  this  report;  these  are  absolute  performance 
character izat ion  tests,  hypothesis  tests,  and  analysis  of  factor-effects  tests. 
Ah so  late  performance  characterization  tests  provide  estimates  of  the  values  of 
selected  performance  parameters  under  a  single  factor  combination  without 
.  era i d-'rat  ion  of  the  effects  of  performance  factors  or  other  performance 
v  As  implied,  the  results  of  such  tests  are  used  to  characterize 

;  'rmance  in  absolute  terms.  Hypothesis  tests  also  reflect  performance  under 
single  factor  combination,  but  the  purpose  of  such  tests  is  to  compare 

performance  with  expected  performance  (or  previously  stated  values) 
r'i-vr  thar.  *  o  characterize  performance  in  absolute  terms.  The  results  of  such 
-■  •  •  m  ty  used  to  support  decisions  concerning  product  improvement 
peir.g  net/not  met  or  performance  meets/does  not  meet  expected 
f  * — .  .  Ir.  analysis  of  factor-effects  tests,  performance  is  compared  for 

.  ,  1  factor  combinations  to  examine,  for  example,  the  "sensitivity"  of 

.  .  ..  ...  ,  *-f  .>-r.ance  to  particular  factors  and  levels  for  those  factors.  Such 
f  1 1  in  support  of  decisions  concerning  system  cost  effectiveness. 

.  .  '.  '  n'.  step  in  the  test  design  process  is  to  select  the  parameters  to 

:•  ,rel.  All  or  a  subset  of  the  parameters  defined  in  Section  5  (Table  H) 

-•iv  ■•••  m-*i  - ;•••'!  during  a  test.  The  principal  constraints  that  influence  the 
■  :f  parameters  to  be  measured  are  measurement  time,  available  data 

-r-’i-fl  n  -,nd  reduction  facilities,  and  data  reduction  costs. 

ird  step  is  to  define  the  population  of  trials  on  which  the  tests 
f -eus.  Tno  following  items  of  information  must  be  specified  in  defining 

•  r  ■  :  ;1  if  lor.  trials  for  <’ach  test: 


•  r.  i  r  a  ■:>  t  er  i  s  t  i  e  s  of  radio  signals  constituting  the  radio 

• ; .  ronrnent  of  tne  SUT 


.•  <t  ion  pc-;  ad'’  r-  '■  during  which  tests  will  he  conducted 


•-•it :  t,  cf  *•  r.e  r,oue  /;;y stem  i n ter f ace (  s )  to  be  monitored 
•  '"./stem  interface(s)  at  which  data  (events)  will  be 


t  r.r'-f:  l<*s  that  define  the  event  sequences,  for 
t.-t  i  ]  r ,  *uit  occur  at  the  use- /system  interface(s)  .at 

'■  i  ;  •  if" i*  no : \ ■  ] r"* •  >1 


sa 
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reference  events  that  correspond  to  each  defined  interface  event 

time-outs  and  thresholds  that  distinguish  successful  trials  from 
performance  failures. 

The  population  must  be  defined  in  such  a  way  that  each  trial  can  be  given  equal 
consideration  and  weight  to  avoid  bias  in  the  estimation  of  population  param¬ 
eters.  "Equal  consideration  and  weight"  often  are  achieved  by  random  sampling, 
however,  the  cost  implications  of  random  sampling  may  limit  the  population  that 


car.  he  used  for  a  test. 


fourth  step  in  the  test  design  process  is  to 


the  factors,  the 


or  values'  for  each  factor,  and  the  factor  combinations  to  be  tested. 


■al,  comprehensive  list  of  relevant  factors,  levels,  and  factor  combina- 
■  .»r.  be  defined,  because  the  appropriate  factors,  levels,  and  factor 
ions  depend  on  the  specific  system  under  test  and  the  objectives  of  the 


t  v  D  i  c  a 


factors  and  associated  levels  for  electronic 


r  systems  are  listed  in  Table  5. 

relection  of  factors,  levels,  and  factor  combinations  in  a  test  should 

'•y  the  following  principles: 

rerforr.ar.ee  factors  and  levels  should  be  distinguished  in  a  test 
design  only  if  their  effects  must  be  specifically  determined  to 

i cr. ;•••/*  the  test  objectives.11 


defined  factor  combination  should  be  tested  at  least  once, 
the  entire  test  should  be  replicated  to  identify  significant 

o ecu r. ted  factors. 


When  the  numbo”  of  defined  factor  combinations  is  too  large  to 
permit  testing  of  each,  the  tested  factor  combinations  should  be 
: ho son  so  as  to  provide  maximum  accuracy  in  comparing  factor 
levels  whose  effects  are  expected  to  be  most  important.  In 
general ,  tno  selected  factor  combinations  should  include 
comb  in  it  ions  that  differ  only  in  these  critical  factor  levels. 

.  whi  'h  '■-very  possible  combination  of  the  defined  factor  levels  is  used 


termed  a 


oil  factorial  test. 


A  test  in  which  some  of  the 


are  not  used  is  termed  a  fractional  factoria  1 
•itify  interactions  among  factors  is  lost  in  a 
The  impact  of  this  diminished  factor  interaction 
e  examined  -as  part  of  the  test  design  process. 


'mb ‘.nation  is  usr-d  in  absolute  performance 


ir, :  .  • 


> 

» 


// 


Table  5.  Some  Typical  Performance  Factors  and  Levels  for  Testing  Electronic 
Surveillance  Systems 


PERFORMANCE  FACTOR 

TYPICAL  LEVELS 

Radio  signals  that  constitute 
test  environment 

the 

A  number,  n^  ,  that  represents  a 
nonstressed  environment 

A  number,  t\2,  that  represents  a 
marginally  stressed  environment 

A  number,  ng,  that  represents  a  highly 
stressed  environment 

strengths  of  radio  signals  that 
constitute  the  test  environment 

Strengths  that  represent  nearby  loca¬ 
tion,  intermediate  location,  and 
distant  location  for  each  signal  source 
A  matrix  of  n  signal  sources  and  3n 
signal  strengths  results  from  which  an 
appropriate  subset  of  signal  sources 
and  strengths  are  chosen 

'  7  jpor  itir.g  Mode 

Manual 

Automatic 

Panoramic 

'  "  v:  ••:  i  ,-r  :>r.sitivity 

Threshold 

Threshold  +  10  dB  (for  example) 

:  Fre.: uvr.-y  Selection 

Discrete  Frequency,  or 

Scan  Single  Band  (or  parts  thereof),  or 
Scan  All  Bands  (or  parts  thereof) 

.'"T  ,  :.‘ss  ing  and/or  Display 

Time 

Specification  Requirement  Two  Times  the 
Specification  Requirement  (for  example) 

The  fifth  step  ir.  the  test  design  process  is  to  select  a_  representative 
sy.p  of  or formar.ee  trials  from  the  defined  population.  The  basic  consider- 
:;e>?ti.ng  performance  trials  to  form  the  test  sample  is  randomization, 
f  •  trials  selected  should  constitute  a  random  sample  of  the  popula- 
t.  r, .  F'.r  a  comogeneous  population,  this  is  achieved  when  each  performance 
'•  .  .  ;  •  -,r.  chance  of  being  included  in  the  sample.  A  second  considers” 

:  •  f  .'••ting  test  samples  is  sample  size.  Sample  size  may  be  derived  from 


precision  objectives  or  specified  on  the  basis  of  practical 
such  as  data  storage  capacity  or  a  reasonable  duration  for  the 
tter  whether  measurement  precision  objectives  or  practical  con- 
used  to  determine  sample  size,  a  desired  confidence  level  or 


up 


significance  level  for  the  test  results  should  be  specified  in  the  test  design. 
Confidence  level  is  a  numerical  value,  typically  expressed  as  a  percentage, 
that  defines  the  likelihood  that  a  confidence  interval  calculated  from  the 
sample  data  will  contain  the  true  value  of  the  estimated  parameter. 
Significance  level ,  which  is  the  complement  of  the  confidence  level,  is  the 
corresponding  specification  in  hypothesis  testing.  Confidence  levels  of  90  or 
9o  percent  ^corresponding  to  significance  levels  of  10  or  5  percent)  are  used 
common ly.  In  general,  the  desired  precision  in  a  test  should  be  determined  by 

cost  of  conducting  the  test  and  the  potential  impact  of  the  resultant  data. 

The  sixth  step  is  to  specify  a.  factor  combination  for  each  test. 
v  ■ -.surement  accuracy,  clearly  defined  applicability  of  the  measurement  results, 

:  efficient  use  of  test  resources  were  noted  earlier  among  the  general 
:tiv.-s  of  a  good  test  design.  The  objective  here,  then,  is  to  define 
:  .  ••  nr:-:- inaf  ions  '.test  conditions)  that  achieve  a  favorable  combination  of 

1.;  t  accuracy  and  efficient  use  of  test  resources.  In  all,  factor 

•.•.mi  :n  tti  ons  should  be  assigned  to  each  test  as  randomly  as  possible  under  the 
'ur.sf' ilr.ts  of  each  test.  For  example,  measurement  efficiency  may 
r  randomization  ir.  some  situations,  due  to  the  additional  time  and  cost 
■. -s '  ' :  :t‘- :  *ith  setting  up  a  particular  factor  combination  repeatedly,  rather 
>•.  f  -  i : '  tests  that  may  use  the  same  factor  combination. 

:•  r. ,  .  ,  the  seventh  step  in  the  test  design  process  is  to  describe  the 

•  ■  •-  i ,;u  witr.  an  explio  it  mathematical  model,  if  possible.  Such  a  model 

:  ■  .-I:--;-  a  concise  synopsis  of  the  test  design  and  a  basis  for  estimating 

•  :  :  r*  precision  and  performance  variability  in  the  data  analysis.  Simple 

it .  :  il  models  oftftn  may  be  used  to  relate  measured  and  statistical 
;  nr.*.  i*  ies  such 

an  observed  value  of  the  factor  in  question 

the  true  [ tut  unknown)  population  value  of  the  factor 

factor  effects  observed  in  the  tests 


index  errors. 


v  i lee::-  of  the  factors  may  be  expressed  as  a  function  of 
‘hi':',.  However,  in  the  case  of  absolute  performance 
avpothesis  tests,  factor  effects  are  not  considered  and 
;v  1  ’.  -  w  i  n?  s  1  mp  L  i  l »  d  f orm : 


i 

V  v  ■r.  *r.  . 

•*,  l<*M  4.'  •“  .  /  _ 

1  «-l 

I  i  =  u  +  e 


Y is  the  value  measured  in  the  i-th  observation, 

u  is  the  parameter's  true  (population)  mean,  and 

e i  is  the  experimental  error  in  the  i-th  observation. 

Juch  -i  model  might  be  used,  for  example,  in  describing  a  measurement  of  signal 
:•>*  "tier,  time  or  system  identification  and  location  time  in  an  absolute 
pe”f.'rma:i:'e  eh  iracterization  test  of  an  electronic  surveillance  system. 

Performance  factors,  such  as  shown  in  Table  5,  may  or  may  not  be 
i  f:  ;■>!•  .  Tests  involving  several  levels  of  a  single,  nonquantif  iable 


deled  by  tn  equation  of  the  form 


value  measured  in  the  i-th  observation  at  factor 


is  the  parameter's  true  (population)  mean, 

is  the  performance  effect  of  a  particular  factor  level  j, 


experimental  error  in  the  i-th  observation  at 


•  arm  quantifiable,  the  factor  effects  can  be  described  with 

h-1  of  the  form 


i  •  :,  x  ••  * 


:  tne  v  of  the  dependent  variable  measured  in  the  i-th 

:  a-.e'vaticn, 

■  not  int  'the  intercept  of  the  regression  line) , 

:  •  .•nctur.t  '  the  slope  of  the  regression  line), 

vil  of  the  <  quanti  f  ianle)  factor  level  in  the  i-th 
i "  i  -o ,  ,nd 

to*  ••••<*  •  i’.  e”r--'r  in  the  i-th  observation. 


t/W-V.  V  .  <  r 


•CXvN-Nv •WW'WW'V 


Lr.V 


Trie  use  of  a  mathematical  model  in  describing  performance  measurements  is 
recommended.  Further  information  on  the  use  of  mathematical  models  in  test 
(experiment)  design  is  available  in  Cox  (1958). 

6.2  Data  Collection 

Tests  conducted  in  accordance  with  the  concepts  described  in  this  report 
require  tnat  certain  raw  performance  data  be  collected  at  the  source/system  and 
use” -'system  interfaces.  Estimates  of  the  functional  performance  parameter 
v uluru  tro  calculated  from  these  raw  performance  data  in  accordance  with  the 
procedures  described  in  Sections  6.3  and  6.4.  We  use  the  concept  of  a  generic 
or f  ;  monitor"  as  the  mechanism  for  obtaining  these  data.  In  practice, 
r-.:  '  interface  monitor  would  include  both  hardware  and  software  components 
■'  *.  C  ’llect  the  data  and,  in  general,  would  be  unique  to  each  monitored 

-ir.  1  the  system  being  tested.  In  the  context  of  Figure  1,  this 
r"  is  represented  as  the  MOBILE  S«JT  INTERFACE  UNIT  and  the  SYSTEM 
.':"E  -N  lEr.F A'E  APPLIQUE.  The  interface  monitor  must  perform  three  major 


-  c  t  Interface  Events.  Detection  and  i  n  terpre  ta  t  i  on  of 
ir  a:  :rred  signals  are  interface  events.  Each  event  must  be 
•..is  *‘>i  a  ted  with  a  time  of  occurrence,  or  "time-stamped." 

•'  ; E  ve  n  t  s .  The  time-stamped  interface  events  are  system 
m.'Vifi:  data.  These  system-specific  interface  events  must  be  mapped 
i-.t  •  system-independent  reference  events  in  accordance  with  the 

'.•v.-., mi- i-.qy  described  in  Sections  4  and  S. 

■: e  -■ ", r  1  f-arerice  Events .  Reference  events,  with  associated  time  of 

•r. mesne,  are  recorded  in  a  performance  data  file(s). 

f  tvs*  functions  is  discussed  in  the  subsequent  paragraphs.  It  should 
t  '  *.-.•*  same  applications  may  be  relatively  simple  while  other  applica- 
at  ivelv  -romp lex,  depending  on  the  system  being  tested  and  the 
f  i  .-’es  of  f  motional  performance  that  are  used  in  testing  the 

r.  ; f  this  methodology  will  need  to  interpret  and  restrict  the 
•  :  f  *  •  i  ,  ‘  d-t’.ogv  accordingly. 

.  i  •  *  i  n  sn.-'-ss  begins  with  the  collection  of  interface  events 
<ri  i  ■- -'system  interfaces.  Each  element  of  information 
;• ‘  •'cf  i--r  is  defined  us  an  interface  event.  Several  types  of 
•  ■  :  .  interfaces  are  illustrated  in  Figure  37.  In  a 
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a)  Source/System  Interface  with  Human  Operator 
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b)  Source/System  Interface  with  Computer  Control 
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Program)  End  User 


Figure*  37.  Simple  illustrations  of  typical  source/system  and  user/ 
system  interfaces  (physical  and  functional  boundaries). 


controlled  environment,  the  source/system  interface(s)  is(are)  physical  or 
functional  boundary( ies)  between  the  system  under  test  and  the  source(s)  of  rf 
signals  and  system  control  commands  to  which  the  SUT  is  expected  to  respond. 
The  user/system  interface  is  a  physical  or  functional  boundary  between  the  SUT 
and  "users  of  information"  produced  by  the  SUT.  Physical  boundaries  are 
illustrated  by  Figure  37a,  37b,  and  37c;  a  functional  boundary  is  illustrated 
by  Figure  37d. 

The  interface  monitor(s)  must  detect  all  signals  transferred  (in  either 
direction)  across  these  interfaces  during  the  test  period,  determine  the  time 
of  occurrence  for  each  of  these  events,  and  interpret  the  transferred  signals 
and  associated  times  of  occurrence  into  a  sequence  of  discrete  events  that  each 


has  specific  meaning. 


The  "users  of  information"  (sometimes  termed  the  end 


i sens'  produced  by  the  SUT  may  be  human  operators  of  the  system,  a  computer 
ippli cation  program  that  utilizes  the  information,  or  a  recording  device  that 
will  store  the  information  for  later  use.  Typical  recording  devices  are 
electronic  memory  devices,  magnetic  tape,  magnetic  disks,  and  a  variety  of 

information  printing  devices. 

Ine  interface  events  are  system  specific  and,  therefore,  cannot  be  used 
li^eotly  in  expressing  system-independent,  user-oriented,  performance 
parameters.  This  problem  was  anticipated  in  defining  the  measures  of 
functional  performance  in  terms  of  system-independent  reference  events. 
Trerefore,  it  is  necessary  to  translate  the  system-dependent  (specific) 
interface  events  into  system-independent  reference  events.  This  translation 
o  inabi  lity  of  the  interface  monitor  is  the  second  function  in  the  data  collec¬ 
tion  process.  Table  6  lists  reference  events  for  the  three  functions  of  an 
electronic  surveillance  system  and  provides  brief  discussion  of  the  signifi¬ 
cance  of  these  events.  The  table  contains  no  system-specific  interface  events, 
..it  indicates  by  the  blank  column  for  listing  those  events  that  test  planning 
f  r  each  specific  system  needs  to  include  identification  of  the  interface 
events  that  are  to  be  monitored  at  the  source/system  and  user/system  interfaces 
so  translated  into  the  3ystem-independent  reference  events.  The  reference 
•  f  r  electronic  surveillance  systems  defined  in  Table  6  are  briefly 

:  i S’ -cis sol  in  the  following  paragraphs. 

Search  Command  could  he  any  of  a  number  of  actions  (that  would  cause  an 
inner f  tee  eventl  that  correlate  with  the  start  of  a  test  and  begins  the 


detection  time. 


Conceptually,  an  instruction  from  the  test 
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controller,  manually  or  automatically  initiated,  or  an  action  such  as  turning 
on  the  SiJT  or  turning  on  the  signal  generators  that  produce  the  test  signals  or 
initiating  a  new  operational  mode  for  the  SUT  could  correspond  to  this 
reference  event. 

Indication  of  Detected  Signal,  likewise,  may  correspond  to  any  of  a  number 
■  f  interface  events.  If,  for  example,  the  SUT  incorporates  a  narrow-band 
receiver  that  is  operating  in  a  swept  frequency  mode,  this  reference  event 
j'.  :  correspond  to  an  indication  of  rf  energy  that  exceeds  an  rf  noise 
f  i  le.g.,  a  spectrum  analyzer-type  CRT  display),  an  indication  of  a 

c.-rn’ in:'  frequency  on  a  user's  or  test  control  monitor,  or  some  other  character 
r . r.g  Slivered  to  tne  monitor. 

Verify  Indication  of  Detected  Signal  (i.e.,  compare  measured  carrier 
f re  ;  :nr.  ?y  with  Known  carr  ier  frequency)  is  the  reference  event  that  determines 
i ’  '>'!  iet-rct  ion  function  outcome  is  successful  or  unsuccessful  (false 
.  This  reference  event  also  stops  the  counting  of  signal  detection 
time.  detention  time,  then,  is  the  difference  between  the  time  of  occurrence 
f  r  t:  1  ••  -event  and  the  time  of  occurrence  for  the  Search  Command  event. 

"Time-Out"  Without  Evidence  of  Signal  Detection  occurs  when  the  SUT 
attempt  to  detect  the  test  signal  is  not  complete  within  a  time  interval  that 
rv.y  no-  specified  by  the  system  user  or  test  controller.  Nonperformance  for  the 
.UT,  f  :;r  the  detection  function,  is  the  function  outcome  when  this  reference 
event  is  recognized.  This  event  would  end  the  trial.  The  ratio  of  nondetec¬ 
tions  tc  the  total  number  of  detection  attempts  (opportunities)  would  indicate 
the  r.ondetection  probability. 

Character ization  Command  is  the  reference  event  that  starts  the  count  for 
signal  character ization  time.  This  event  may,  in  fact,  be  coincident  with 
com: let  ion  of  tne  detection  function,  or  it  may  be  an  explicit  instruction  from 
the-  user  or  test  controller.  Processing  of  the  test  signal  will  provide  pulse 
wictn  and  pulse  repetition  frequency /interval .  The  functions  of  detecting  a 
.  gr.a  1  end  processing  the  signal  to  determine  pulse  width  and  pulse  repetition 
f'’e  vim. i  nterval  are  concept  ully  straightforward  when  considering  narrow¬ 
er.;,  ’.-frequency  receiving  systems.  This  conceptual  process,  however,  may 

net  -c  ir  so  straightforwardly  in  broad-band  receiving  systems  in  which  the 
■v- :  sur-.:nent  of  ea^r  frequency,  pulse  width,  and  pulse  repetition 
1  ;  c '  lcte”val  may  be  an  i  nte.gr  a  I  process. 
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Indication  of  Test  Signal  Pulse  Width,  conceptually,  is  the  first  step  in 
character ization  of  the  measured  test  signal.  Indication  of  pulse  width  may  be 
provided  to  the  user's  or  test  controller's  system  monitor  and  thus  would  be 
available  as  an  interface  event  that  can  be  monitored  as  an  intermediate 
indication  of  the  signal  characterization  function  outcome. 

Indication  of  Test  Signal  Pulse  Repetition  Frequency/Interval  (PRF/PRI), 
conceptually,  is  the  second  step  in  character  ization  of  the  measured  test 
signal.  Again,  indication  of  pulse  repetition  frequency/interval  may  be 
provided  to  the  users'  or  test  controller's  system  monitor  and  thus  would  be 
available  as  an  interface  event  that  can  be  monitored  as  a  further,  intei — 
mediate  indication  of  the  signal  characterization  function  outcome.  As  noted 
earlier,  the  process  of  measuring  carrier  frequency  and  pulse  characteristics 
may  be  an  integral  process  in  some  systems. 

Verify  Characterization  of  the  Test  Signal  (i.e.,  compare  measured  pulse 
width  and  pulse  repetition  frequency/interval  with  known  characteristics  of  the 
test  signal)  is  the  reference  event  that  determines  if  the  signal 
characterization  function  outcome  is  successful  or  unsuccessful.  Favorable 
comparison  of  measured  and  known  test  signal  characteristics  confirms  success¬ 
ful  performance;  unfavorable  comparison  of  measured  and  known  characteristics 
indicates  incorrect  performance  (incorrect  signal  characterization).  This 
reference  event  stops  the  counting  of  signal  characterization  time.  The  signal 
characterization  time,  then,  is  the  difference  between  the  time  of  occurrence 
for  this  event  and  the  time  of  occurrence  for  the  Characterization  Command. 

"Time-Out"  Without  Indications  of  Signal  Characterization  occurs  when  the 
SUT  processing  of  the  detected  signal,  to  determine  signal  characteristics,  is 
not  complete  within  a  time  interval  that  may  be  specified  by  the  system  user  or 
test  controller.  Nonperformance  for  the  SUT,  for  the  signal  characterization 
function,  is  the  function  outcome  when  this  reference  event  is  recognized.  The 
ratio  of  signal  noncharacter izat ions  to  the  total  number  of  signal  character¬ 
ization  attempts  (opportunities)  would  indicate  the  probability  of  signal 
noncharacterization. 

Emitter  Identification  and  Location  (EIL)  Command  is  the  reference  event 
that  starts  the  count  for  emitter  identification  and  location  time.  This  event 
may,  in  fact,  be  coincident  with  completion  of  the  signal  characterization 
function,  or  it  may  be  an  explicit  instruction  from  the  user  or  test  control¬ 
ler.  The  event  identifies  the  initiation  of  the  EIL  function.  This  function 
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may  be  relatively  simple  or  relatively  complex,  depending  on  the  SUT.  For  less 
sophisticated  (perhaps,  ground-based)  types  of  systems,  the  function  comprises 
the  measurement  of  lines  of  bearing  from  single  locations  and  the  comparison  of 
these  measured  LOB  (and  the  measured  and  calculated  signal  characteristics  data 
from  the  preceding  function)  with  stored  data  for  known  emitters.  For  more 
sophisticated  types  of  systems,  which  may  include  airborne  subsystems,  the 
function  involves  the  measurement  of  lines  of  bearing  from  multiple  locations, 
the  calculation  of  emitter  location  as  the  intersection  of  lines  of  bearing 
that  have  been  measured  from  multiple  locations,  and  the  comparison  of  these 
measured  and  calculated  LOB/location  data  and  the  measured  and  calculated 
signal  characteristics  data  (preceding  function)  with  stored  characteristics 
and  locations  for  known  emitters.  (Emission  characteristics  and  geographic 
locations  of  known  emitters  are  stored  in  the  memory  of  computer-based 
electronic  surveillance  systems.)  When  the  measured/calculated  data  compare 
favorably  with  the  stored  data  for  known  emitters,  the  system  ha3  performed 
satisfactorily.  When  the  measured/calculated  data  do  not  compare  favorably 
with  the  stored  data,  system  performance  may  still  be  satisfactory  if  the 
measured/calculated  data  are  for  a  new  (unknown)  emitter.  However,  performance 
is  unsatisfactory  if  the  measured/calculated  data  are  for  a  known  emitter.  In 
the  SLF  test  environment,  signal  character istics  and  "apparent  locations"  would 
be  part  of  the  known  test  conditions  that  could  represent  either  known  or  new 
( unknown)  emitters,  a  condition  controlled  by  the  test  controller.  For  systems 
such  as  the  TEAMPACK  Assembly,  the  typical  process  would  involve  use  of  signal 
characteristics  and  the  measured  LOB  data.  Alternatively,  for  systems  such  as 
the  Advanced  QUICK  LOOK,  the  typical  process  would  involve  use  of  the  signal 
characteristics  and  the  measured/calculated  location  data. 

Indication  of  Line  of  Bearing  to  the  Emitter  is  the  first  step  in 
"identifying  and  locating"  an  emitter.  (Recall  that  the  signal  characteriza¬ 
tion  function  has  preceded  this  function.)  Data  that  define  the  line  of 
bearing  may  be  provided  to  the  user's  or  test  controller's  system  monitor  and, 
thereby,  may  be  available  as  an  interface  event  that  can  be  monitored  as  an 
initial  indication  of  the  emitter  identification  and  location  function  outcome. 
In  the  case  of  systems  like  the  TEAMPACK  Assembly,  this  interface  event  would 
indicate  completion  of  that  portion  of  the  EIL  function  pertaining  to  "location 
determination. " 
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L  ration  of  Emitter  Identification  would  be  the  next  step  in 
•'  and  locating"  the  "unknown"  emitter  (or  the  test  signal).  The 
.  f  emitter  identification  that  is  provided  to  the  user's  or  test 
er's  system  monitor  would  be  available  as  an  interface  event  that  could 
is  an  intermediate  indication  of  the  emitter  identification  and 
?t  ion  oatcc"  .  Measurements  of  signal  characteristics  (preceding 
no:  './  •  cation  data  for  known  emitters  that  compare  favorably  with 
m  i  in..::.  to  a  satisfactory  performance  outcome.  Measured  data  for 
t'  i '  not  compare  favorably  with  stored  data  indicate  an 
•  s  .nee  si.*  '-'me.  fiowever,  measured  data  for  new  emitters  should 
st . "-?d  data,  and  the  performance  outcome  is  indeterminate 
i  su*  it  factory  or  incorrect.  The  available  information  is 

•  ••  •  -  .:-terrn:  In  the  case  of  less  sophisticated  electronic 

the  TSAMPACK  Assembly,  this  interface  event  would 
.  •  ;  .  cf  *.h?  EIL  function  and,  thus,  stop  the  counting  of  emitter 

.  .  I  •- ration  time,  which  would  be  computed  as  the  difference 

o-v.’  r-ror.ce  for  this  event  and  the  time  of  occurrence  for 

•'  o mm  uni . 

ication  of  Emitter  Location  is  the  final  step  in  identifying  and 
•.  ■; :  wr; "  ••m:  t  ter  (or  test  signal).  The  more  sophisticated 

11  in  •••  systems  will  include  the  capability  for  measuring 
■  --v ’:>•  from  multiple  measurement  locations  and  applying 
•  ,  .  j  to  determine  emitter  location  as  the  intersection  of 

.  .  ;  the  limits  cf  some  elliptical  error  probability  that  may  be 
:  •  -.•sf  controller.  The  Advanced  QUICK  LOOK  is  an 

'•  .  ; y ;  •  r  ss/ston.  Tne  indication  of  emitter  location  that  is 

••>•••  test  controller's  system  monitor  (or  an  output 


be  available  as  an  interface  event  that  could  be 


at  ion  of  the  emitter  identification  and  location 
•meat  •/,.vt  1  ou  1  at  ion  of  emitter  ( identification  and) 
1 1.  •  tr:  it  cone  ares  favorabiv  with  stored  data 


nance  outcome. 
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stored  lata  indicates  an  incorrect  performance 
•mi tter  ( identif ication  and;  location  for  a  new 


:  :  ore, 


npf-* 


/  V  v  .* 


outcome  is  indeterminate — it  may  be  either  desired  or  incorrect.  In  the  case 
of  sophisticated  electronic  surveillance  systems  like  the  Advanced  QUICK  LOOK, 
this  interface  event  would  identify  completion  of  the  EIL  function  and,  thus, 
stop  the  counting  of  emitter  identification  and  location  time,  which  would  be 
computed  as  the  difference  between  the  time  of  occurrence  for  this  event  and 
the  time  of  occurrence  for  the  EIL  Command. 

"Time-Out"  Without  Indications  of  Emitter  Identification  and/or  Location 
occurs  wh-->n  the  Sl'T  attempts  to  identify  and  locate  the  "unknown"  emitter  (or 
".•••s*.  signal'  are  not  complete  within  a  time  interval  that  may  be  specified  by 
the  system  user  or  test  controller.  In  other  words,  one  or  more  of  the 
preceding,  repaired  reference  events  for  the  EIL  function  has  not  occurred. 
Nonperf ornar.ee,  for  the  emitter  identification  and  location  function,  is  the 
f  ion  outcome  when  this  reference  event  is  recognized.  In  the  SLF  testing 
; system,  the  interface  monitor  would  require  additional  logic  to  distin¬ 
ction  measurements  for  new  emitters  (that  should  not  compare  favorably  with  the 
stored  ditu:  incorrect  measurements  or  time-out  indicating  no  measurements. 

‘•lest,  but  net  all,  of  the  observed  interface  events  will  relate  to  the 
t primary  functions.  Some  interface  events,  however,  will  translate  into 
two  reference  events,  in  which  case  the  second  reference  event  may  relate  to 
seeoninry  function  of  system  operability  state  (available/unavailable). 
T'nm  aggregation  of  successive  primary  function  times  for  desired  performance 
an:  a -'Cp table  incorrect  performance  and  nonperformance  trials  comprise  the 
time  that  the  system  is  in  an  available  state,  whereas  the  aggregation  of 
successive  primary  function  times  for  nonperformance  trials  comprise  the  time 
that  the  system  is  in  an  unavailable  state. 

The  third  function  of  the  interface  monitor,  noted  earlier,  is  to  record 
the  reference  interface  events.  There  actually  are  two  elements  of  information 
to  it  sh  vjid  he  recorded.  These  are: 

’.  »  uy'.p lete  set  of  reference  events  relating  to  both  primary  and 

secondary  T.L'T  functions  and  the  performance  significance 

"I  r  r.  *  *  -i  wi-.r.  'Vn  of  these  events  and 

tin--  of  jr^ence  (absolute  or  relative)  of  each  reference 
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0.3  Data  deduction 

.•  t  i  or.  describes  the  functional  requirements  for  a  data  reduction 

will  transform  the  performance  data  collected  by  the  interface 

'  estimates  of  the  measures  of  functional  performance.  The  process 
t order. ,  primary,  reference  event  data  to  estimate  values  for  the 
net  era  i_-  described  first.  Then,  the  procedures  for  developing  the 
'  t--*r  data  n*'e  described.  Some  applications  of  these  procedures 

tv-  is-  Imp:--'';  but,  in  general,  the  discussion  assumes  off-line 

:  ••  er-.-r.oe  event  information.  The  data  reduction  procedures 

fur.c "  i-'hal  flow  charts,  outcome  diagrams,  and  mathematical 


used  in  this  section  are  shown  in  Table  7 .  Several 
•  :  in  defining  these  symbols  are  listed  below. 

represented  by  a  lowercase  mnemonic  svmbol 
e-t-.-otion,  "c"  for  signal  characterization,  and 
1  lent  i  f  irat  ion  and  location'.  The  various 
omes  are  represented  by  subscripted  lowercase 
for  successful  detection,  "cs"  for  successful 

. i  ■'.••it  ion,  and  "13"  for  successful  emitter 

vi  location. 

•f  function  "opoortuni ties"  (less  than  or  equal 
trials'  an  i  function  outcomes  observed  during  a 
i  ay  corresponding  uppercase  letters  ( e. g. , 
number  of  detection  "oonortunlt ies”  and  "D„" 

O 

:•  jT  successful  detections,  "C"  for  the  total 
teri.tat  ion  "opportunities"  and  Cs  for  the  total 
character  izations,  etc. ) . 


...  times  are  represented  by  the  symbol  t(  ). 
ion  performance  times  (one  "opportunity"  per 
rented  by  the  symbol  w(  ).  Probabilities  and 
v*'  tine 3  are  represented  by  the  symbols  PC  }  and 
The  argument  (  )  ir.  e,aoh  case  is  the 
of  interest.  For  example,  the  expression 
■  v-'r, ige  fin-'-  ?or  successful  ietection. 

-.:•••  *  --r  value.-,  ar--  d  i  s  t  i  n  g  u  i  s  h  i  fr-'-rn  f  h- 
.  values  h  /  * no  upper  ••ay-:-  subsm :pt  ■•: .  For 
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user  system  interface  interactions  during  the  trial  being  considered.  (The 
tstal  number  of  consecutive  trials  constitute  the  test.)  The  procedure 
rr 'duces  as  its  output  an  estimated  value  for  each  of  the  three  detection 
performance  parameters,  w(ds),  P(df),  and  P(dn),  and  the  updated  average 

detection  time,  W(ds). 

The  first  step  in  the  data  reduction  procedure  for  the  detection  function 
is  to  initialise  tre  variables  used  in  recording  the  detection  function 
,.t  -ernes.  -inch  detection  "opportunity”  (or  attempt)  is  the  result  of  a  search 
•.'.-•'mar.u  being  issued  manually  by  the  test  operator,  automatically  by  the  SLF, 
automatically  by  the  3M7  as  a  normal  part  of  operation  during  testing.  The 
Starch  command  will  increment  the  detection  "opportunity"  counter,  D,  and  mark 
ire  start  of  detection  time,  t(d). 


Following  issuance  of  the  search  command,  there  is,  conceptually,  a 
gical  a.nd  recurring  check  for  indication  of  a  detected  signal.  If  there  is 
in  lie. it  ion  of  a  detected  signal,  the  elapsed  time  in  attempting  to  detect 
e  signal  is  checked  to  determine  if  the  specified  (required)  performance  time 
tec  tier.,  Wp  i  ds ) ,  has  elapsed.  When  the  elapsed  time,  w(de),  is  less  than 
,  the  logical  process  loops  to  check  again  for  a  detected  signal.  When 
is  equal  to  or  greater  than  WR(ds),  "time-out"  without  detection  has 
•'urred.  Monperformance  for  that  detection  "opportunity"  is  the  outcome,  and 

■  >  r.cn detection  outcome  counter,  Dn,  is  incremented  by  one.  The  data  reduc- 
n  procedure  for  that  trial  is  ended,  and  the  procedure  re-starts  with  the 

iijrs  for  estimating  detection  function  parameter  values  for  the  next 

If  trere  is  an  indication  of  a  detected  signal,  the  procedure  checks  some 
^'arte-ishic  'e.g.,  the  carrier  frequency)  of  the  detected  signal  for  a 
wit-,  the  known  value  of  the  signal  characteristic.  When  this  "match" 
-  negative,  the  logical  conclusion  is  false  detection,  and  the  incorrect 

iri-e  counter,  Dp,  is  incremented  by  one.  The  data  reduction  procedure 

■  v.  if  t-i  el  is  ended,  and  the  procedure  re-starts  with  the  procedure  for 
•  •  d'-tert, ion  function  parameter  values  for  the  next  trial.  Desired 

r-.vir.o-;  r.as  occurred  when  the  "match”  check  is  positive,  and  the  successful 
v  o  outcome  count''--,  0S,  is  incremented  by  one.  The  end  of  the  detection 
‘'s.- ,  also  is  recorded. 

In-,  ri-'xt  step  in  t-ie  procedure  is  to  calculate  the  measured  values  of  the 
'  .-tier:  performance  parameters,  w(ds),  P(dp),  and  P(dn) ,  and  to  update  the 
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estimate  of  average  detection  time,  W(ds).  The  detection  function  performance 
outcomes  and  equations  for  calculating  the  estimates  of  parameter  values  are 
illustrated  in  Figure  39.  The  procedure  for  estimating  detection  function 
performance  parameter  values  now  is  complete,  and  the  data  reduction  procedure 
advances  to  the  procedure  for  estimating  the  performance  parameter  values  for 
the  signal  characterization  function. 

The  procedure  for  estimating  parameter  values  for  the  signal 
characterization  function  is  shown  in  Figure  MO.  The  procedure  assumes  as  its 
inpu*-  a  sequence  of  recorded  primary  reference  events  that  represent  the 
source/system  and  user/system  interface  interactions  and  that  successful 
detection  has  occurred  for  the  trial  being  considered.  (The  total  number  of 
consecutive  trials  constitute  the  test.)  The  procedure  produces  as  its  output 
in  estimated  value  for  each  of  the  three  signal  character ization  performance 
parameters,  w(cs),  P(cf),  and  P(cn),  and  the  updated  average  signal 
^r.aracterization  time,  W(cs). 

The  first  step  in  the  data  reduction  procedure  for  the  signal 
character ization  function  is  to  initialize  the  variables  used  in  recording  the 
signal  characterization  outcomes.  Each  signal  characterization  "opportunity" 
or  attempt)  is  the  result  of  a  character ization  command  that  is  issued,  either 
manually,  automatically  by  the  SLF,  or  automatically  by  the  SUT,  but  it  must 
follow  a  successful  detection  function  outcome  for  that  trial.  The  character¬ 
ization  command  will  increment  the  character ization  "opportunity"  counter,  C, 
anl  mark  the  start  of  the  signal  characterization  time,  t(c). 

Tnore  is  a  conceptually  logical  and  recurring  check  for  indication  of  the 
measured  signal  character ist ics  (frequency,  PW,  and  PRF/PRI)  following  issuance 
of  th“  characterization  command.  If  there  is  no  indication  of  a  complete  set 
-,f  measured  signal  characteristics,  the  elapsed  time  in  attempting  to  charac- 
t’.-'iz"  the  signal  is  checked  to  determine  if  the  specified  (required) 
>:T,” time  for  signal  characterization,  WR(cs),  has  elapsed.  When  the 

•  !  tiro  ,  w'  cf.) ,  is  less  than  WR(cs),  the  logical  process  loops  to  check 
,  f  ,  ■  ;  v  leto  measured  signal  characteristics.  When  w(ce)  is  equal  to  or 
•re:*;,.'  t.r.un  Wr,/  -\.) ,  "time-out"  without  complete  signal  characterization  has 
cure!.  Nonperformance  for  that  signal  characterization  "opportunity"  is  the 
.*.  ,  o.  :  '  onuha^acter ization  outcome  counter,  Cn>  is  incremented  by  one. 

;  ■  •  :  jet  ion  procedure  for  that  trial  is  ended,  and  the  procedure 


DETECTION 

PARAMETER  EQUATIONS 

1.  Detection  time  (trial) 

=  w(df)  =  t(d#)-t(d) 

2.  Average  detection  time 

=  w Z  w(ds) 

*  ds=  1 

3.  False  detection  probability 

Df 

4.  Nondetection  probability 


DEFINITIONS 

dt=  Successful  detection. 

D=  Total  number  of  detection 
"opportunities"  during  the  test. 

Df=Total  number  of  false 

dectections  during  the  test. 

DnsTotal  number  of  nondetections 
during  the  test. 

D*=  Total  number  of  successful 
detections  during  the  test. 

t(d)  =  Start  time  for  the  detection 
function. 

t(<^)*Stop  time  for  successful 
detection. 


Figure  39.  Detection  function  performance  outcomes  and  equations  for 
calculating  estimates  of  the  parameter  values. 


restarts  with  the  procedure  for  estimating  detection  function  parameter  values 
for  the  next  trial. 

If  measured  signal  characteristics  are  indicated,  the  procedure  checks  the 
measured  characteristics  for  a  "match”  with  the  known  character istics  of  the 
test  signal.  When  this  "match"  check  is  negative,  the  logical  conclusion  is 
incorrect  signal  characterization,  and  the  incorrect  performance  counter,  Cp, 
is  incremented  by  one.  The  data  reduction  procedure  for  that  trial  is  ended, 
the  procedure  re-starts  with  the  procedure  for  estimating  detection 
function  parameter  values  for  the  next  trial.  Desired  performance  has  occurred 
wn-sn  too’  "match"  check  is  positive,  and  the  successful  signal  characterization 
outcome  counter,  Cs,  is  incremented  by  one.  The  end  of  the  signal  character- 
iz  at  ion  time,  t(cs),  also  is  recorded.  (This  part  of  the  data  reduction 
lure  for  the  signal  characterization  function  could  be  expanded,  if 
:*-r  to  account  for  partial  completion  of  function.  We  have  assumed  that 

function  is  either  completed  or  not  completed.) 

The  next  step  in  the  procedure  is  to  calculate  the  measured  values  of  the 
•  i.--.  il  characterization  performance  parameters,  w(es),  P(cp),  and  P(cn),  and  to 
up  late  the  estimate  of  average  signal  characterization  time,  W(cs).  The  signal 
"har voter izat ion  function  performance  outcomes  and  equations  for  calculating 
•he  "at imstes  of  parameter  values  are  illustrated  in  Figure  41.  The  procedure 
for  estimating  signal  character ization  function  performance  parameter  values 
now  is  complete,  and  the  data  reduction  procedure  advances  to  the  procedure  for 
e  5 ti mating  the  performance  parameter  values  for  the  emitter  identification  and 
'O' a*  i  or.  f unction. 

The  procedure  for  estimating  parameter  values  for  the  EIL  function  is 
no  wr.  in  Figure  ^2.  The  procedure  assumes  as  its  input  a  sequence  of  recorded 
reference  events  that  represent  the  source/system  and  user/system 
into” face  interact  ions  and  that  successful  signal  characterization  has  occurred 
f  •”  the  trial  being  considered.  (The  total  number  of  consecutive  trials 
. not . t  it?  hue  tost,  but  it  is  unlikely  that  all  functions  of  every  trial  will 
:  oo'V  the  signal  characterization  and  EIL  functions  will  not  be 
-.o'  -  -  tod  if  th-c  c receding  function  has  not  been  completed  successfully.)  This 
•  urr  :  i  an  it-.  output  an  estimated  value  for  each  of  the  three  EIL 

'  ”•  ’  "•  •••'•  ;  ir’Jt-d'-m,  w  i  ; '  ,  P  ( 1  e ) ,  and  P(ln),  and  the  updated  average  EIL 


INCORRECT 
CHARACTERIZATION 
\  PROBABILITY 
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NON¬ 
CHARACTERIZATION 
PROBABILITY  / 
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SIGNAL  CHARACTERIZATION 
PARAMETER  EQUATIONS 

1  Characterization  time 
(trial)  =  w(cs)  =  t(cs)  —  t(c) 

2. Average  characterization 
.  C. 

time  =  W(c,)=  —  £  w(cs) 
c#=1 

3. Incorrect  characterization 

Cf 

probability  =P(cf)=  — 

4.  Noncharacterization 

Cn 

probability  =  P(cp)  = 


DEFINITIONS 

c,  =  Successful  signal  characterization. 

C=  Total  number  of  signal  character¬ 
ization  "opportunities"  during 
the  test. 

Cf  =  Total  number  of  incorrect  signal 
characterizations  during  the  test. 

Cn=Total  number  of  noncharacter¬ 
izations  during  the  test. 

Cs= Total  number  of  successful  signal 
characterizations  during  the  test. 

t(c)  =  Start  time  for  the  signal 
characterization  function. 

t(cs)  =  Stop  time  for  successful 
signal  characterization. 


a  1  i  !;  tr.icUr  i?  it  ion  function  performance  outcomes  and 


> ; 1 1  =  1 1  i .  >  n.  s 


for  r.iliui  -aim;  i-st  [nates  of  tlu  parameter  values. 


CONTINUE  FROM 
PROCEDURE  FOR 
CHARACTERIZATION 
PARAMETERS 


INITIALIZE 
Ell  PERFORMANCE 
PARAMETERS 


CALCULATE 
MEASURED  VALUES 
OF  EIL 
PERFORMANCE 
PARAMETERS 


Procedure  for  estimating  emitter  identification  and 
location  function  parameter  values. 


The  first  step  in  the  data  reduction  procedure  for  the  emitter 
dentif icat ion  and  location  function  is  to  initialize  the  variables  used  to 
eoord  the  EIL  outcomes.  Each  EIL  "opportunity"  (or  attempt)  is  the  result  of 
EIL.  command  that  is  issued,  manually  by  the  test  operator,  automatically  by 
he  SLF,  or  automatically  by  the  SUT;  but,  it  must  follow  a  successful  signal 
maracter ization  function  outcome  for  that  trial.  The  EIL  command  will 
cerement  the  EIL  "opportunity"  counter,  L,  and  mark  the  start  of  the  emitter 
fixation  and  location  time,  t(]_). 

There  is  a  conceptually  logical  and  recurring  check  for  indication  of  an 
■mitter's  identification  and  location  from  the  measured  data  (frequency,  PW, 
>:di-V  ?R1 ,  and  LOB  from  a  reference  location  or  the  location  coordinates)  that 
'■'.lows  tr.e  issuance  of  the  EIL  command.  If  there  is  no  indication  of  the 
:o‘--y’minah  ion  of  emitter  identification  and  location  from  the  measured  data, 
;•:■  el  ip.se  i  time  in  attempting  to  determine  emitter  identification  and  location 
•i  „■•  ••  to  determine  if  the  specified  (required)  performance  time  for 

:mitter  identification  and  location,  Wr( 13) ,  has  elapsed.  When  the  elapsed 
1  me ,  w"  1.3' ,  is  less  than  Wp(ls),  the  logical  process  loops  to  check  again  for 
?.mitter  identification  and  location  determination  from  the  measured  data.  When 
; '  l ,, '  is  equal  to  or  greater  than  Wr(13),  "time-out"  without  determining 
•mifer  identification  and  location  has  occurred.  Nonperformance  for  the  EIL 
’  is  the  outcome,  and  the  EIL  nonperformance  outcome  counter,  Ln, 

. •'.•cited  by  one.  The  data  reduction  procedure  for  that  trial  is  ended, 

r  •■  ..-ceduro  r (-.-starts  with  the  procedure  for  estimating  detection 
:n  ;  irameter  values  for  the  next  trial. 

I  f  •  mi‘.t-r  identification  and  location  are  indicated  from  the  measured 
:  i'  t,  the  ch-'cks  the  "measured"  identification  and  location  for  a 

'■••••  ••-."  »i‘-  -he  -r.'wr.  emitter  identification  and  location  for  the  test  signal. 

"“-th.-h"  re-v  is  negative,  the  logical  conclusion  is  incorrect 
■  :  .  c  ••>  f  •■  *•..  ‘  i'.  •*  ion,  and  the  incorrect  performance  counter,  Lf ,  is 

;  ‘  ,  -'••••duct ion  procedure  for  that  trial  is  ended,  and 
•  -  •  :•■*.  wi ‘me  procedure  for  estimating  detection  function 
.■  :  -  *  -•  ‘  t.  —  i a  1 .  Tesired  per formance  has  occurred  when  the 

■  -  ■  ‘  .  ,  and  tr<-  successful  emitter  identification  and 

■  ’  ■  •"  •’■  ■'.  -  a  in'"'  em-nt^d  by  one.  The  end  of  the  emitter 

l.-o  is  recorded.  (This  part  of  the 
d-  r.t  i  f  1  c a t  i or.  and  locati on  function 


a 


■j  1  ■  -  be  expanded,  if  desired,  to  develop  unique  procedures  for  systems  that 
termine  only  line  of  bearing  as  opposed  to  systems  that  fully  determine 
cation  as  the  intersection  of  at  least  two  lines  of  bearing  from  different 
Terence  locations.) 

The  next  step  in  the  procedure  is  to  calculate  the  measured  values  of  the 
itter  identification  and  location  performance  parameters,  w(ls),  P(lp),  and 
I.;',  and  to  update  the  estimate  of  average  emitter  identification  and 
■■  -a  -  i  n  time,  w'ls'.  The  emitter  identification  and  location  function 
rf'.'rr  tr.  ■•••  outcomes  and  equations  for  calculating  the  estimates  of  parameter 
ire  ; :  lust  rated  in  Figure  43.  The  procedure  for  estimating  emitter 
i  on  and  location  function  performance  parameter  values  now  is 
•r  ir.d  the  trial  is  ended.  The  data  reduction  procedure  advances  to  the 

"''id  i r. ■:  starts  the  procedure  for  estimating  the  performance  parameter 
tr.e  defection  function. 

procedures  discussed  so  far  provide  estimates  of  values  for  the 
»?•>'  performance  parameters.  Values  for  the  secondary  performance  param- 
estimated  following  similar  procedures.  Consider  now  the 
.  1 1  it-  i  1  i  ty  ’’unction  that  maps  primary  performance  outcomes  over  performance 
•  • .  : inf  ■  system  operability  states.  In  general,  this  process  requires 
t  i or.  ■'  the  primary  parameters  to  be  used,  definition  of  the  performance 
Ids  f "  these  parameters,  and  specification  of  the  performance  period  to 
d  in  making  the  logical  decision  concerning  system  operability 


■  fue  following  parameters  to  facilitate  discussion  of  the 

:\oi  t  i  on  and  performance  parameters.  These  parameters  provide  the 
n.-f  at  ion  to  discuss  a  test  that  spans  T  performance  periods,  with 

nr :  r.g  each  performance  period. 

:  n  r.fea  to*  ifh  trial  during  the  jth  performance  period, 

:  -  ’  ,  ?.,  •  •  •  T  and  j  =  1  ,  2,  •  •  •  T, 

:  'he.o  a  3  rr'-jssful  f"ial, 

.  •  -  r-..  juiced  fraction  of  successful  trials  during  a 

:  '  •'  ih  '-'  i'vi , 


Hi  or* 


of  trials  in  the  jth  performance 


t  ' 

•2 

tj 


SUCCESSFUL 
EMITTER  IDENTIFICATION 
AND  LOCATION 


(TRIAL) 


TIME  w(£s) 


a$) 


AVERAGE 

EIL 

TIME  W(5LS) 


INCORRECT 

EIL 


INCORRECT 

EIL 

PROBABILITY 

PUf) 


EIL 

NONPERFORMANCE 


El  L  / 

NONPERFORMANCE  / 

PROBABILITY  /  / 

// 


EIL  PARAMETER  EQUATIONS 

1.  EIL  time  (trial) 

=  w(Hs)  =  ttts)-t(M 

2.  Average  EIL  time 

.  *-• 

=  W(jL)=~L  Z  w(£J 

LiV« 

3. Incorrect  EIL  probability 

iP(V!r 

4. EIL  nonperformance 

L 

probability  =  P(&n)  =  — 


DEFINITIONS 

£#  =  Successful  emitter  identification 
and  location. 

L=  Total  number  of  EIL 

"opportunities"  during  the  test. 

Lf  =  Total  number  of  incorrect  EIL's 
during  the  test. 

Ln=  Total  number  of  EIL  non¬ 
performances  during  the  test. 

L8=  Total  number  of  successful 
EIL's  during  the  test. 

t(fc)  =  Start  time  for  the  EIL  function. 

t(£,)  =  Stop  time  for  successful  EIL. 


-it  cor  identification  and  Location  function  performance 
■  i*  .  .  -m  •=  and  equations  for  calculating  estimates  of  the 
*  rare  ter  "  > ! UeS . 
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denotes  the  total  number  of  successful  trials  in  a  performance 
period, 

denotes  the  jth  performance  period  of  the  test, 
denotes  a  successful  performance  period, 

denotes  the  minimum  fraction  of  successful  performance  periods 
required  for  successful  (satisfactory)  system  operation, 

denotes  the  total  number  of  performance  periods  in  a  test, 

denotes  the  total  number  of  successful  performance  periods  in  a 


•’  :;i'  denotes  the  probability  of  a  successful  trial  during  a 
performance  period,  and 

p  :3’  denotes  the  probability  of  a  successful  performance  period 
during  a  test. 

Toe  basic  element  of  a  test  is  the  trial,  b.  Three  primary  functions 
~ c h ion  id),  signal  characterization  (c),  and  emitter  identification  and 
t  ion  '1''  will  be  performed  during  each  trial  when  the  system  operates 
~f  uteri  I y.  If  a  primary  function  is  not  performed  successfully,  however, 
trial  is  ended,  the  remaining  functions  are  not  attempted,  and  a  new  trial 
ns.  'erne  number  of  trials,  8,  based  on  achieving  statistical  significance 
■.  test  results,  will  constitute  a  performance  period,  t,  for  the  purpose 
clVtg  the  system  operability  state.  A  threshold,  bps,  will  be  defined  by 
test  planner  for  the  fraction  of  successful  trials  that  are  required  during 
ccessfui  performance  period,  that  is,  Bs/3  i  bps>  (We  require  all  func- 
s  to  be  completed  successfully  for  a  successful  trial.  Logically,  this 
iremont  means  that  only  the  emitter  identification  and  location  function 
he  checked  for  successful  completion,  since  each  function  of  the  trial  is 
ted  only  if  the  preceding  function  is  completed  successfully.  In  other 
s,  .my  function  completed  incorrectly  or  not  completed  causes  an 
•  •-ssf j:  vial.)  The  relationships  of  these  events  (not  to  be  confused 

tv-  interface  events  that  are  discussed  in  other  sections  of  this  report) 
-  “ i  up seb  tenting  time  associated  with  these  events  are  illustrated  in 

re  ' Figure  21  illustrates  the  concept  of  successful  and  unsuccessful 

Vn  V"!  by  "1"  or  "0"  respectively,  during  successive  performance 
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SECONDARY  PARAMETER  EQUATIONS 
l.TTT  =  t(rT)-t0 

2.SPPT=  [t(r,)-t(r,  .)] 

J  J  J  R  /R>  h_ 


B  /B>  b0 

»  R* 


3.TSPT=i[t(r)-t(r.  )] 

S,/B>b„f 

4  Availability,  A  =  (TSPT)/(TTT) 


5.  OTBF  = 


Tf-H 


( TSPT) 


6.  FPPT  =  [t(r.)-t  (*-._,)] 
'f  k«1  ' 

,(th’]  c 
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DEFINITIONS 
b  =  A  test  trial. 

bR#=  The  required  fraction  of  success¬ 
ful  trials  during  a  performance 
period. 

B=  The  total  number  of  trials  during 
a  performance  period. 

Bf=  The  total  number  of  successful 
trials  during  a  performance  period. 
FPPT  *A  Failed  Performance  Period 

_  (elapsed)  Time. 

FPPT*The  average  Foiled  Perform¬ 
ance  Period  (elapsed)  Time 
(typical  MTTR). 

k  =  The  counting  index  for  failed 
performance  periods;  i.e.  k=1,2...Tf 
OTbF- The  average  Operating  Time 

Between  Failures  (typical  MTBF) 
SPPT  =  A  Successful  Performance 
Period  (elapsed)  Time, 
t  =  Test  time. 
t0=Test  start  time. 
f(r.)s  Time  that  the  j  th  performance 
J  period  is  ended;  j  =  1,2...T. 

T  s  The  total  number  of  performance 
periods  comprising  a  test. 

Tf  =  The  total  number  of  failed  perform¬ 
ance  periods  during  a  test. 
TSPT=The  Total  Successful  Perform¬ 
ance  (elapsed)  Time. 

TTT  s  Total  Test  (elapsed)  Time. 
t  =  A  performance  period. 


fivstem  operability  state  (secondary)  function  performance 
outcomes  and  equations  for  calculating  estimates  of  the 
parameter  values. 
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rtit'r.  the  basis  above  for  describing  system  performance,  it  is 
straightforward  to  define  the  probability  of  a  successful  trial,  bs,  during  a 

performance  period,  i,  to  be 

?(bs)  =  3S/B,  and 

"■'.■j  probability  of  a  successful  performance  period,  i5,  during  a  test  to  be 

O'-  \  -  -r  /  T 

■  '  ■  S  •  ~  ‘S'1- 

isf  i-tory”  system  perTormance,  then,  can  be  defined  as  the  condition  when 

?'•  ts'  ^  VT  -  THs- 

It  is  important  to  note  that  these  definitions  deal  with  the  probabilities  of 
ev-'.nts  incurring  and  that  time  is  only  an  implicit  parameter.  Equal  numbers  of 
•c'iil?  or  performance  periods)  do  not  necessarily  represent  equal  periods  of 

‘ .  t  '  ir  system  operation''  time. 

Though  the  event  probabilities  as  descriptions  of  system  performance  for 
the  second. -.ry  function  are  conceptually  straightforward,  it  is  not  convenient 
w  i  *  r  tr.ese  definitions  to  define  operability  states  for  the  system,  that  is, 
ti">:  when  the  system  is  available  and  time  when  the  system  is  unavailable  and 
times  for  transitions  between  these  states. 

In-'  concepts  of  reliability  and  availability  frequently  are  used  to 
:  /el  p  *!  mere  general,  more  macroscopic  description  of  system  performance  than 
./lied  by  the  definitions  of  the  primary  performance  parameters.  A  common 
tefiniti on  of  reliability  is  the  probability  that  a  system  will  operate  without 
failure  for  a  specified  function  (or  above  some  thresholds)  for  a  specified 
iod  of  time.  As  noted  earlier  in  this  discussion  of  secondary  performance 
i--  cm*--  ter  s ,  this  definition  of  reliability  does  not  distinguish  between 
incorrect  performance  and  nonperformance,  but  combines  these  two  outcome 
mrco  into  the  single,  more  general  category  of  performance  failure. 
A /  i i  1  as l  .  i ty  j? ten  is  defined  as  the  probability  that  a  system  will  be  in  an 
;  -.t  i  -'h  at  any  arbitrary  time  during  some  much  longer  test  (or  field 

c  •  :  '•vp-.n.'ntial  model  for  characterizing  system  operability 
•  t  :  i'-  ::  :  a  I",  v  in  .lection  U .  The  key  issues  are  to  choose  the 

*  •’  :  .  i  an’ to  be  jsed  to  measure  availability  performance, 
tr,-  ririm.r  -v  =  .■.  r-‘*ment  time  over  which  the  selected  function 
:  .<  "■  !.',jred,  and  to  specify  the  threshold  values  to  be 
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used  in  defining  system  failures.  We  consider  a  successful  trial  to  occur  when 
the  emitter  identification  and  location  function  is  completed  successfully 
(which  means  that  detection  and  signal  characterization  functions  also  have 
been  completed  successfully).  Minimum  measurement  time  will  be  defined  by  the 
test  planner  consistent  with  the  statistical  quality  desired  from  the  test 
results.  Section  7  of  this  report  contains  guidance  for  making  that  decision. 
Equations  for  calculating  estimates  of  the  secondary  function,  system  perform¬ 
ance  parameter  values,  with  time  as  an  explicit  parameter,  are  given  in 
F i cure  •’* ,  along  with  the  illustrations  of  performance  outcomes. 

6.4  Data  Analysis 

This  section  discusses  methods  for  analyzing  the  measured  system 
per:'  '  cm a see  data  and  describes  statistical  information  that  should  be  prepared 
•ar. :  reported  with  the  measurement  results.  Analysis  methodology  and  statis¬ 
tical  information  that  correspond  to  each  of  the  three  general  types  of  tests 
that  are  identified  in  Section  6.1  (and  for  which  the  approach  followed  in  this 
r-.-por1-  is  useful),  namely,  absolute  performance  characterization  tests, 
hypothesis  tests,  and  analysis  of  factor-effects  tests,  are  presented.  The 
data  analysis  methods  are  described  only  to  the  extent  necessary  to  define  the 
minimum  requirements  for  reporting  measurement  precision.  The  subject  of 
st  itistical  data  analysis  is  addressed  comprehensively  in  other  reports  and 
.  d-:s  (e.g.,  Fisz,  1963,  or  Hoel,  1962).  The  application  of  statistical  data 
analysis  techniques  to  digital  communication  systems  is  developed  thoroughly  in 
the  report  by  Miles  (1984). 

Absolute  performance  characterization  tests  are  performed  to  characterize 
the  performance  of  an  electronic  surveillance  system  under  a  single  specified 
set  of  conditions  (a  particular  factor  combination)  without  concern  about 
•'  i':t.,r  effects  or  previously  stated  performance  values.  Such  tests  are 
intended  to  be  used  .n  estimating  population  parameters  from  sample  data;  they 
prov i do  n ,  basis  for  decisions  based  on  performance  comparisons.  A  parameter 
••.-.;t  '.’Vito  on  leu  la  ted  from  measured  data  cannot  be  expected  to  equal  exactly  the 
true  population  value  because  of  sampling  error.  Therefore,  it  is  important 
f  >r  such  an  estimate  to  he  accompanied  by  an  explicit  specification  of  measure¬ 
ment  precision.  The  primary  purpose  of  the  data  analysis  in  absolute 
;  e-' 'ermines  n.iructerizut  ion  tests  is  to  develop  this  specification. 


The  precision  of  a  population  parameter  estimate  calculated  from  a  finite 
sample  is  expressed  in  terms  of  a  confidence  interval  and  an  associated 
confidence  level.  A  conf idence  Interval  is  a  range  of  values  about  a  measured 
parameter  estimate  within  which  the  "true"  (population)  value  of  the  parameter 
can  be  expected  to  be,  with  a  stated  confidence  (in  percent).  The  end  points 
of  a  confidence  interval  are  called  confidence  limits.  These  limits  may  be 
expressed  either  in  absolute  terms  (e.g.,  ±1.0  min)  or  in  relative  terms  (e.g., 
half-length  of  the  confidence  interval  divided  by  the  estimate). 

Confidence  level  is  defined  in  Section  6.1  as  a  numerical  value,  typically 
expressed  as  a  percentage,  that  defines  the  likelihood  that  a  confidence 
interval  calculated  from  the  sample  data  will  contain  the  true  value  of  the 
estimated  parameter.  If,  for  example,  a  95  percent  confidence  level  is 
specified,  confidence  intervals  calculated  from  individual  samples  will  contain 
the  "true"  parameter  value  in  about  95  out  of  100  samples.  Figure  ^5 
illustrates  a  set  of  20  (hypothetical)  calculated  confidence  intervals, 
96  percent  confidence  assumed,  with  one  interval  that  does  not  include  the  true 
r  d  r  _i  :ri'3 1  o  ”  me  a  n . 

Methods  far  calculating  90  and  95  percent  confidence  intervals  for  digital 
.'omnunioation  system  parameters  are  described  in  detail  in  the  report  by  Miles 
' '  .  These  or  equivalent  methods  should  be  used  in  calculating  confidence 

intervals  for  all  absolute  performance  characterization  tests  that  are 
:•  :nd  acted  on  electronic  surveillance  systems  following  the  test  approach 
lev* loped  in  tnis  report. 

hypothesis  testing  is  testing  from  which  the  validity  of  a  particular 
statistical  hypothesis12  is  examined  and  ultimately  accepted  or  rejected.  A 
st at  istlcal  hypothesis  is  an  assumption  about  the  distribution  of  a  population, 
expressed  in  terms  of  specified  values  for  one  or  more  population  parameters. 

simple  hypothesis  testing  is  described  here  in  which  a  performance  value 
measured  under  a  single  factor  combination  is  compared  with  a  previously 
spoMf led  hypothetical)  value  to  determine  if  a  "significant"  difference 
:  V  ',.  The  decision  to  accept  or  reject  a  hypothesis  normally  is  made  with 
:  -•  ur.  'ortainty,  since  the  parameter  estimate  based  on  a  finite  sample  can 


*  hypothesis  traditionally  is  called  the  null  hypothesis,  because 

.to  f  vi--.  hypothesis  implies  that  no  difference  exists  between  the 

and  ' population  values. 
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•deviate,  sometimes  substantially,  from  the  true  parameter  value.  The 
uncertainty  cf  a  hypothesis  test  is  expressed  by  its  significance  level ,  a, 
which  is  the  probability  of  rejecting  the  tested  hypothesis  when,  in  fact,  it 
is  true .  1 3 

The  hypothesis  to  be  tested  and  a  desired  significance  level  must  be 
specified  during  test  design.  The  data  extraction  and  data  reduction  processes 
produce  an  estimate  of  the  measured  parameter.  With  these  inputs,  an  analysis 
to  test  the  hypothesis  that  a  specified  value  equals  the  true  population  mean 

is  accomplished  as  follows: 

1.  Calculate  a  confidence  interval  from  the  measured  data  following 
the  methods  that  have  been  described  above.  If  the  null  hypothe¬ 
sis  is  true,  the  calculated  confidence  interval  will  include  the 
specified  value  with  probability  equal  to  (1  -  a). 

1.  Compare  the  specified  value  with  the  confidence  interval.  If  the 
specified  value  lies  within  the  confidence  interval,  the  null 
hypothesis  can  be  accepted  with  a  significance  level  (probability 
of  error)  equal  to  ot.  If  the  specified  value  lies  outside  the 
confidence  interval,  the  null  hypothesis  is  rejected. 

In  some  hypothesis  tests,  the  purpose  may  be  to  determine  if  the  actual 
per  fsrm.ancc  is  equal  to  or  better  than  (rather  than  exactly  equal  to)  a 
specified  value.  The  approach  described  above  can  be  applied  to  such  tests  by 
a  imply  using  half  of  the  significance  level.  The  resultant  value  expresses  the 

in  i  1  i  ty  that  a  measured  value  lies  within  that  part  of  the  confidence 
interval  that  represents  performance  equal  to  or  better  than  the  specified 
v  iluc.  Th''>  same  approach,  can  be  used  to  test  a  negative  hypothesis  (actual 
performance  is  not  as  good  as  a  specified  value). 

In  some-  hypothesis  tests,  it  may  be  necessary  to  consider  another  type  of 
err  o-ror  of  accepting  a  stated  hypothesis  when  it  actually  is  false.  I* 

'.:-:e  linked  of  such  an  error  is  determined  by  three  variables: 

'.  the  significance  level,  a,  of  the  test 
tn-‘  test  sample  size 

;.  -•>  »  tu  >1  difference  between  the  hvnotheti c,al  and  "true"  values. 


•f  reie-^tinv  a  true  hVDOthesis  is  called  a  typo  I  error. 


accepting  'failing  to  reject)  a  false  hypothesis  is  called  a 
The  probability  of  a  type  ! I  error  commonly  is  represented  by 


Specific  relationships  between  these  variables  may  be  determined  using  "power 
curves,"  such  as  the  operating-characteristic  curves  described  in  the  manual  by 
Trow,  Davis,  and  Maxfield  (I960). 

Analysis  of  factor  effects  tests  are  tests  conducted  using  several  factor 
come  ir.at  ions,  and  the  results  are  compared  in  the  analysis  to  identify  and 
quantify  postulated  factor  effects.  The  analysis  typically  consists  of  two 


analysis  o : 


variance  or  an  equivalent  category  of  data 


analysis  to  identify  the  significant  factor  effects  and 


riividual  performance  comparisons  to  examine  and  quantify  such 


of  variance  is  a  statistical  technique  by  which  the  observed 


sample  is  separated  into  several  components,  each  of  which 


■lability  attributable  to  a  particular  factor.  The  variance 


■  :  eacn  factor  is  compared  with  a  residual  variance,  attributed  to 
’ror,  and  the  factor  effect  is  deemed  significant  (at  a  particular 
level,  a)  if  the  variances  differ  more  than  predicted  by  the 
r  statistic.  The  procedure  is  described  in  the  Statistics  Manual 

i ,  and  Max  field ,  1 960 ) . 

•is  of  variance  is  recommended  for  use  in  evaluating  the  effects  of 
factors  or.  til  time  and  rate  parameters  specified  in  this  approach 
electronic  surveillance  systems  using  the  SLF.  An  equivalent 
■f  lata  analysis  that  uses  the  x2  (chi-squared)  statistic  should  be 
he  analysis  of  failure  probabilities.  This  analysis  procedure  also 


ei  in  the  Statistics  Manual.  Formulas  for  calculating  the  x^  and  F 


e  included  in  the  application  of  statistical  analysis  techniques 
mmjni^ation  systems  by  Miles  ( 198*0. 

ir.a lysis  of  variance  (or  an  equivalent  category  of  data  analysis) 


postulated  factor  has  no  significant  effect  on  performance, 


1  ’.eve Is  of  the  factor  may  be  combined.  This  combining 


•forman.ee  specification  by  eliminating  unnecessary 


When  significant  factor  effects  are  identified, 


•  a^i sons  normally  are  undertaken  to  examine  those 


serve  two  ob iectivea: 


vs  > 


’ .  tj  simplify  the  specification  as  described  above  by  identifying 
particular  levels  of  a  performance  factor  that  need  not  be 

distinguished  and 

to  provide  a  basis  for  defining  quantitative  relationships 
between  factor  levels  and  performance  values. 

Per  f  nrmanoe  data  for  different  factor  levels  may  be  combined  whenever  one 
measured  value  lies  within  the  confidence  interval  of  another. 

The  most  direct  way  to  summarize  quantitative  relationships  between  factor 
levels  and  performance  values  is  simply  to  list  the  calculated  values  (sample 
'-■'•.ns  an:  •jr.fidence  limits  for  each  level.  These  data  also  may  be  graphed  in 
,  cloic  „'iys  to  present  possible  models  of  relationship. 

r:n*ire  are  three  additional,  more  formal  data  analysis  and  presentation 
m-  :t  may  be  used  to  provide  more  detailed  information  about  a  measured 

:  ;  .lit  .  T so  methods  are: 

aphid  a!  presentation  of  frequency  distributions 

•'  , 'harts 

r-.vrej.si  ?n  analysis. 

T • . a e  tr.”-' -  methods  apply,  respectively,  to  the  three  general  types  of 
p-*r’f  tests  that  we  have  been  discussing. 

1”  i -a  I  presentations  of  frequency  distributions  include  diagrams  such 
us  hist 'grams  and  cumulative  distributions.  Control  charts  are  graphical 
pres c  1-os  of  systematic  variations  in  a  monitored  process.  Regression 
v.  >lvsi-a  is  •;  mathematical  method  for  expressing  relationships  between  random 
.■e'ir  les,  i.e. ,  performance  factors  and  parameters.  Each  of  these  techniques 
•  ..  .-?•>,»  i-e  *  in  trie  references  for  statistical  methods  that  have  been  cited 

’.  ••'KASi'REMENT  METHODS  FOR  TYPICAL  ELECTRONIC  SURVEILLANCE  SYSTEMS 
'•>  -t ion  •  presents  a  structured  approach  to  the  description  of  system 
•-  .  Applying  the  structured  approach  in  Section  5  to  two  typical 
••  n;  survei  1  ler.ee  systems  leads  to  the  definitions  of  11  measures  of 
.  •  •  i  -  si  r  ep  f  ;rm  ;n 'e  that  relate  to  3  primary  functions  and  1  secondary 
*  ;  n.;t>-  m  i  ■  ■  surveillance  systems.  (The  structured  approach  applied 

•  -  f  sv  -feme  would  follow  the  same  process,  but  other  functions 

o  of  f  met ior.al  performance  would  be  defined.)  The  performance 
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n.  approach  tnat  encompasses  a  four-phase  testing  process  is  define.1 
*" .  Mow,  in  Section  7  specific  measurement  methods  are  discussed  for 
election:-  surveillance  systems  using  the  SLF  and  other  testing 
ios  of  the  Electromagnetic  Environment  Test  Facility.  First,  an 
for  a  Detailed  Test  Plan  for  electronic  surveillance  systems  is 
Turn,  specific  test  modes  (SLF  tests,  computer  simulation,  bench 
i  facility  t-'-ots,  etc.)  and  the  interrelationships  of  these  various 
o  d  "l  tsed. 

’ .  ’  at  line  for  a  Detailed  Test  Plan 
Art./  regu  1  t*  ions  that  define  the  life  cycle  (development, 
*  .  id4,  goer  ;M '  t  •  1  use'  of  Army  C-E  equipments  and  systems  identify  a 
'if-.'  '.--s*  o  Development  Tests  at  various  phases  of  develop- 

f*. str  ‘  various  phases  of  development,  First  Article  Tests, 

•  .  '•  '  nr; 'ted  before  the  equ i pment /system  can  be  certified  for 

.r-r.^r.r  an  1  operational  use.  The  Detailed  Test  Plan  (DTP) 

•  . ::  ,  resented  in  Table  S  is  intended  for  general  applicability; 

.  .  •  :•/  be  r.e'.nss  >.ry  to  tailor,  slightly,  this  outline  before  using 

;  .  ai  i  T-sst  Plan  for  a  particular  type/phase  test.  It  will 

test  clan  outline  covers  SLF  tests,  bench  tests  (in 
-  c  :  — t  t.cd'or  the  antenna  test  range),  computer  simulation 


lined  during  bench  tests),  and  Field  Facility  tests, 

•  st  '•heck-outs  of  the  system,  subsystem,  or  eompon- 
>;-d  outline  for  The  Detailed  Test  Plan  is  presented 


lest  Mode  Interrelationships 

trie  use  of  user-oriented  (system  independent) 
evaluate  and  describe  the  performance  of  sophisti- 
would  be  tested  using  the  SL11'.  In  developing  and 
’’or  these  usee-oriented  performance  measurements, 
•-  to  the  testing  of  electronic  surveillance  systems 
censor’s  technical  representative  for  t.cis  study), 
•dance  with  the  methodology  developed  in  this  report 
tests  that  may  involve  the  measurement  of 


p, o ^  *  ^~nane °  o i  r 


:■  x  n. c  v  *;  a  s  i 4 


ijja 


Table  3.  Detailed  Test  Plan  Outline,  (Type/Phase)  Test  of  (Nomenclature  of 
Test  Item) 


INTRODUCTION 


DETAILS  OF  SLF  TESTS 


Pretest  System  Check-out 
. ’ . ’  Object i ves 

Criteria  Appropriate  Regulation) 
C . 1 .  -  D  ata  Required 

Data  Acquisition  Procedure 
C' .  1  . c  Analytical  Procedure 


Detection  Function 
.  T . '  Objectives 


'.0  oVS 


Criteria  Appropriate  Regulation) 

Data  Required 

Data  Acquisition  Procedure 
Analytical  Procedure 


r.al  Character izat ion  Function 
. 1  Objectives 

.  :  Criteria  'Appropriate  Regulation) 
.  ?  Data  Repaired 

Data  Acquisition  Procedure 
.S  Analytical  Procedure 


ter  Identification  and  Location  (EIL)  Function 

1  Objectives 

2  Criteria  (Appropriate  Regulation) 

3  Data  Required 


Data  Acquisition  Procedure 


Analytical  Procedure 


cm  Operability  State  Function  (Secondary) 

1  Objectives 

2  Criteria  (Appropriate  Regulation) 

3  Data  Required 

■1  Data  Acquisition  Procedure 
5  Analytical  Procedure 


INSTRUMENTED  WORKSHOP  TESTS  (BENCH  TESTS) 


eat  System,  Subsystem,  or  Component  Check-out 

■  Objectives 

Criteria  (Appropriate  Regulation) 

3  Data  Required 

Data  Acquisition  Procedure 
5  Analytical  Procedure 


i 


iver  Characteristics  Tests 
!  Do jeet ives 

2  Criteria  Appropriate  Regulation' 

3  Data  Required 

u  Data  Acquisition  Procedure 

-  Analytical  Procedure 

nna  Subsystem  Character ist ics  Tests 

1  ,'b  je-'tives 

Criteria  Appropriate  Regulation) 

Data  Required 

Da-i  Acquisition  Procedure 
Ar.j  ly  t  i  m  1  Procedure 

,  t  ics  Tests 

•  ;t)jcct:v->:- 

'rireri:  Appropriate  Regulation) 

'  ">“’;u  i  rod 

; t  .  A  •  :  .  idition  Procedure 
An  * lyt i'll  Procedure 

:■  'IPCTr/r  .'I VCLATION 

rf  input  Data 

1  t;  i-'-Ct  i  V“.’ 

:ri‘«ri •  Appropriate  Regulation) 

1  it!  Pe pi i red 

1  at  a  i  ■'u;  is  it  ion  Procedure 
.•.:h1v‘  ;  v»l  Procedures 

-  v.ti  r:  zr  t-  'onp uter  Simulation 

,  ’  ;  ;  fj  ‘‘  \  v-S 

.  'rite”!  *  ' Appropriate  Regulation) 

.3  Data  Required 
.  Da- a  Acquisition  Procedure 
.  Ar.aly"icai  procedures 

y  FltLD  r A  I > j T TY  TESTS 

•■-st  C •/ stem,  Subsystem,  and/or  Component  Check-outs 

.  1  -  *y-  '  *  [  703 

.  -  'r: tori  a  'Appropriate  Regulation) 

.  -  '  ata  Required 

.  .  :  : a  a  •'.oouisi  tion  Procedure 

A t  1 1  v  t :  -■  - 1  Procedures 

’."ih  of  to-',  o-.'ld  Facility  Tests 

<  '  A r  nropr i a te  Regulation) 

1  a  n  if’-'  ! 

■  ’  a  A-'  :u is  ;t  ion  Procedure 


••/  tc  verify  that  a  system  is  in  "normal  operating  condition" 
rt  if  any  SLF  testing. 

.g  will  continue  to  be  an  important  component  of  performance 
>F  systems,  subsystems,  and  discrete  equipment  components, 
-ast  two  important  reasons  for  the  importance  of  bench  tests: 

sting  nay  produce  user-oriented  performance  results  that 
be  understood  without  the  results  of  some  bench  testing. 

s  ibsys- -,m  ■  component  performance  requirements  often  will  be 
.el  in  terms  of  engineer ing-or iented  performance  parameters 
ill  mve  t.  be  measured  using  bench  tests.  Many  of  these 
ar :  r:g— or  iented  parameter  values  will  be  required  as  input 
v  -> simulation. 

aim. lari'-,  also  will  continue  as  an  important  component  of 
ion  Army  Z-E  systems,  subsystems,  and  discrete  equipment 
are  it  '.east  two  situations  under  which  computer  simulation 
important.  These  are: 

np  >-»:<  it/  tr  it  is  required  for  adequate  evaluation  of  some 
?  may  ■  <  '•  :—d  the  capabilities  of  the  SLF. 

a*  sim;liried  computer  simulation  can  serve  as  a  good 
,.-r.  for  defining  the  environment  that  should  be  specified 

*  *  ip  o  • 

;  1  i  -  y  testing  can  be  the  most  realistic  evaluation  of  the 
,  systc-  mdor  known  and  controlled  test  conditions.  However, 
e  suhj-'tt  to  the  uncertainties  of  variations  in  propagation 
important,  however,  is  the  fact  that  good  field  tests  can 
w,i3  amount  of  equipment  in  addition  to  the  system  to  be  tested, 
an  be  •/«•■"/  long,  and  the  test  costs  can  be  very  high.  Field 
tner^for-3,  should  be  considered  as  the  tests  of  "last  resort." 
tests  should  be  considered  only  when 

test  modes  and/or  computer  simulation  fail  to  answer 
ier.tly  the  questions  being  asked  concerning  performance  of 
-- m  -'ob: g  t e " ted  or 

f.av  tnese  other  test/analysis  modes  are  inadequate 
•  —  th*'  -valuation  of  system  performance  that  is  required. 


i  A6 


s.  :.':jclusi oris  and  rkjdmmendations 

my  Electronic  Proving  Ground  has  extensive  test  capabilities 
leetromagretic  Environmental  Test  Facility  that  are  usee,  tc 
:/EMV  of  d.S.  Department  of  Defense  C-E  systems  and  equipment, 
sophisticated  C-E  systems  and  equipment  are  being  developed, 
est  capabilities  need  to  be  upgraded  substantially  to  ensure 
testing  is  performed  to  assure  satisfactory  operation  of  the 
that  this  testing  is  performed  as  economically  as 
-•  -  ir.g  Facility  is  envisioned  as  a  capability  that  will 


i  *■  '■  trie  development  of  methodology  tc  ut  i  line  the 
that  have  been  .addressed  in  performing  the  study. 

of  existing  f’.,F-type  test  capabilities,  (f )  the 
f  in  -t  ior.al  performance  as  the  basis  for  evaluating 
i  '  surveillance  systems,  and  ■.  V  the  development  of 
:  c  do  1  ogy  f  or  SL  F  u  t  i  1  i at  ion. 

•  hr -type  capabi  1  iti-’S  considers  capabilities  that 
f  i'oAKP-1,  Capabilities  that  exist  within  P  SALTO 


st  "  nap  ah  i  1  i "  i  es  whi'-h  are  identified  and  discussed 
t‘-st  -  ap  a "  ill*  i  es  'which  are  discussed  in 
-.si  v-'.  co".  4*  •  ••  sima!  at  ion  capabilities  (which  are 
;  -n  .  •.  J ' .  M  any  af  these  capabilities  will 
as  tic  'u son  t  :n  .'cation  ? .  For  example,  the 


r.  1  int  it  ed  scenarios;  the  identification  and 
i-'-loym-nt  >-.d  integrated  scenario  that  realisti- 
1  by  the  GHT;  the  development,  updating,  and 
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coupling  techniques,  developed  for  testing  avionics  systems  on  military 
aircraft.  USAEPG  is  planning  to  purchase  an  Advanced  Tactical  Electronic 
a v' fa re  Simulator  that  functionally,  at  least,  will  become  the  major  portion  of 
the  SLF  Non-COMM  Threat  Simulator.  The  Central  Target  Simulator  (at  NRL)  is  a 
large,  state-of-the-art  laboratory  facility  (anechoic  chamber)  that  effectively 
includes  a  TEWES.  The  NRL  facility  is  designed  to  operate  only  over  the 
frequency  range  of  3  to  1 8  GHz.  The  USAEPG  SLF  will  require  a  facility  of  this 
type,  •'  : r  with  substantially  expanded  capabilities  to  accommodate  the  COMM 
Tore  if  Simulator,  as  well  as  the  expanded  frequency  range  of  the  Non-COMM 
Tor*',  it  .limn labor .  The  expanded  frequency  coverage  at  lower  frequencies  will 
r-' ;  appropriate  physical  expansion  of  the  facility,  which  may  be  quite 

cm  b  If  unless  some  direct  or  radiated  near-field  coupling  techniques  can 
;ped  and  implemented  at  the  lower  test  frequencies.  Techniques  for 
n  -r-loTi  "nr  ling  of  rf  energy  have  been  developed  and  are  used  for  testing 
fv  iug  operability)  avionics  systems  on  military  aircraft.  These  techni- 
•  ave  a  number  of  limitations  that  will  be  very  difficult  to  overcome  for 
'  -  *.  ■cst  in.*.  Ear  example,  the  alignment  of  one  antenna  with  respect  to  the 
*  is  v.-ry  critical  to  obtaining  repeatable  results.  These  alignment 
are  me*  through  the  development  and  use  of  elaborate  and  expensive 
::•••/ 1  res  fc"  exact  positioning  of  the  antenna  (surface  conforming  for  the  SUT) 

•  ■  insure  repeatable  test  results.  Another  important  consideration  in  using 
v.-ur-C ‘  'Id  coupling  techniques  is  the  translation  of  test  results  observed  when 
.  ir  s  r.e  jr-f  leld  coupling  to  system  behavior  under  normal  operating  conditions 
fur-field  coupling  would  be  expected). 

major  ureas  of  development  required  for  implementation  of  the  SLF  test 
■  cm. t  seem  to  be 

rom-nr.  of  tne  Non-COMM  Threat  Simulator 

level  op m*--nt  of  suitable  rf  energy  coupling  techniques  for  the 
fu :  :  frequency  operating  range  of  the  SLF 

d--  f :  -  It  ‘-•n  *■  the  physical  enclosure  (anechoic  chamber), 

.no  lot-'''  *  v !  ‘  n  the  two  items  above,  for  the  SLF 

:■  f:>.  ;*  ;  i* •  v ■  cpment  of  the  central  computer  and  test  control 


h  fir  it  ion  development  of  interface  units  for  the  SUTs,  which 
would  include  the  source/system  and  user/system  interfaces  for 
identifying  and  recording  interface  events  in  accordance  with  the 
test  methodology  developed  in  these  studies 

further  refinement  of  the  test  methodology  that  is  developed  in 
the  report  following  the  structured  approach  that  is  system 


rr  O''-1  dur 


■r  measures  of  functional  performance  has  followed  a 
'.he  problem  of  defining  system  functions  and  selecting 
toe  performance  of  the  system.  The  approach  follows 
to  ensure  that  the  selected  set  of  performance  param- 


,  efficient,  and  measurable.  The  parameter  development  is 
e  point:  of  view  of  the  user  who  produces  parameters  that  are 
i  performance  rather  than  measures  of  the  causes  (of 


’Tvr;  within  the  system.  Such  parameters  are  system 
;,  very  useful  for  soecifyir.g  the  performance  requirements 
fled  or  designed  and  for  comparing  performance  among 
•  development  process  involves: 

V-;  interfaces  for  inputs  and  outputs 

trv  f  motions  performed  by  the  system  in  terms  of 
no  ritputs 

0:^3  for  the  primary  functions  (intended 
w'onr- ret  performance,  or  nonperformance) 


‘  o’  *•  •' ameters  of  interest  from  the  matrix  of  all 

1  r.  ”i  mary  function  and  outcome  pair  possibilities. 

:•  r :  e r. t e o  p  ar  moters,  though  system  dependent,  certainly  are 
ft :i”e  ‘-•cs'-nt ial  for  identifying  and  understanding  the  causes 

.  •  •  •  '  .  ,r;  .  ^  ^  £  •  P  f'  f'  '3  0  S  ,  ) 

s -■  a  o  -  r  ■  >  r  provide  descriptions  of  system  performance  during 
:'>o,  but  a  complete  character izat ion  of  performance 


:ription  of  the  frequency  and  duration  of 


"secondary"  parameters  are  defined,  using 
r imary  param- ■*  e- ' ,  *  a  iervr  ifce  system  performance 

f  v  i “w  often  ar-S'v- 1  d  with  the  concept  of 
•.ry  f  jn  -r  ions  .m'1  i  :  !  JStrnt..-.  j  for  several  types  of 


E  systems  that  include  communications  systems,  navigation/timing  systems, 
mote  sensing  systems,  and  electronic  surveillance  systems. 

At  the  request  of  the  sponsor,  emphasis  is  given  to  electronic 
rveillar.ee  systems  in  the  test  methodology  development.  Two  systems, 
nsidered  typical  of  EWI  systems,  are  described.  One  system  is  the  AN/MSQ-103 
ec ial-Furpose  Receiver  Set  (commonly  known  as  TEAMPACK),  which  is  a  ground- 
sod  (small  vehicle-mounted)  intercept  and  direction  of  arrival  system  for 
enti Tying  and  locating  unfriendly  Non-COMM  systems.  The  other  system  is  the 
v '..need  i'JZZK  LOOK  System,  which  can  include  up  to  three  airborne  Non-COMM 
itte1"  identification  and  location  subsystems  that  are  connected  to  a  ground 
t-\  analysis  subsystem  via  wideband  data  links. 

The  structured  approach  to  describing  system  performance  parameters  from  a 

•  perspective  is  applied  to  the  development  of  performance  parameters  for 
l wo  systems  identified  above.  Source/system  and  user/system  interfaces  are 

fin*  d  >•:.(  illustrated  (using  functional  block  diagrams)  for  each  system, 
r  ;e  t i ;  functions,  for  the  general  class  of  electronic  surveillance 
-•  tu ,t*’  include  signal  detection,  signal  characterization,  and  emitter 

•  •  i  and  location  are  used  and  the  possible  outcomes  (desired 

•’ f  ,  incorrect  performance,  and  nonperformance)  are  discussed  and 

;  it1;  :  for  ■  acr.  function.  This  process  leads  to  nine  primary  parameters 

>r-.ary  par-meters  that  are  defined  as  the  recommended  measures  of 
r  .  ...n  ■; :  ;•  erf  or  nance  ft"*  these  ewi  systems.  These  parameters  would  be 

.  .  .  •  •  *  any  ;tr'nic  surveillance  system  for  which  the  primary  func- 

cicual  detection,  signal  character ization,  and  emitter  identification 

I'-.-  ?-vr.p'etu>  process  for  using  the  3LK  to  test  systems  that  would  be 
.  :  •. i  r.,*  the  set  if  measures  of  functional  perf  ormance  developed  for  the 

.  .  :  a.-  ..  ;t--:r.s  then  Is  described.  This  process  includes: 

•  n-o-''edur<‘s  f  ••*  and  steps  to  be  followed  in  developing  a  good 

i  ;*r. 

of  and  methodology  for  developing  interface  monitors 
■I  :  nt<r  face  events,  process  these  events,  and  record 

•••  fi event.;  teat  are  the  data  required  to  determine  system 

-  re  jui-enent:'.  and  procedures  for  reducing  the  reference  event 
•.!,';  into  i  ited  values  for  the  primary  and  secondary 

•  '...•.*  •  n  :;f  tne  msur-'s  of  functional  performance) 


procedures  for  analyzing  the  measured  system  performance  data  to 
determine  (or  assure)  the  statistical  significance  of  the  data. 


§ 


Finally,  specific  measurement  methods  are  discussed  for  testing  electronic 
surveillance  systems  using  the  SLF  and  other  testing  (and  analysis)  capabil¬ 
ities  of  the  EMETF.  First,  an  outline  for  a  Detailed  Test  Plan  is  presented. 
Secondly,  the  interrelationships  of  specific  test  and  simulation  (analysis) 


nodes  are  discussed. 


Several  recommendations,  based  on  the  results  of  this  test  methodology 


Lopment,  are  offered. 


The  structured  approach  to  the  development  of  measures  of 
functional  performance  using  functions  and  parameters  that  are 
user-oriented  and  system  independent  offers  wide  opportunity  for 
specifying  desired  system  performance  in  terms  that  are  meaning¬ 
ful  to  users  and  for  comparing  system  performance  results  using  a 
common  basis;  we  strongly  recommend  the  use  of  these  measures  of 
functional  performance  for  SLF  testing. 


Further  study  is  needed  to  understand  the  relationships  between 
overall  SLF  test  frequency  capabilities,  physical  size  required 
for  the  enclosure,  and  methods  for  coupling  rf  energy  at  all  test 
frequencies  and  evaluate  the  impact  of  these  factors  on  continued 
SLF  development. 


Ir  testing  is  planned  that  will  employ  antenna-to-antenna 
coupling  of  rf  energy  in  the  near  field,  further  study  is  needed 
to  extend  system  performance  results  observed  under  these  near¬ 
field  conditions  to  expected  performance  under  normal  operating 
conditions  (presumed  to  be  far-field  conditions). 
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APPENDIX  A:  DEFINITIONS  OF  TERMS  AND  ACRONYMS 


its  o  w n 


’ield  of  electronic  warfare  (EW),  like  many  other  fields,  ha3  produced 
lexicon.  In  addition,  the  U.S.  Departme*  t  of  Defense  uses  many 


in  their  documents.  This  appendix  is  incli  -  to  assist  readers  in 
; ; r. tetter  the  purpose  of  this  report  by  familiarizing  them  with 
in  the  report.  Some  of  the  terms  defined  are  peculiar  to  EW,  while 
iefir.ed  in  a  way  that  is  unique  to  the  SLF  and  this  report,  e.g., 
r. ■ : r  teen  previously  defined.  In  all  cases,  an  attempt  has  been 


definitions  consistent  with  other  documents  and  as  concise  as 


loss  of  substance, 


Availability  Function,  A(t)  -  The  probability  that  a  system  will  be  in  an 
•  state  at  time  t,  luring  the  total  mission  time. 


Comm. in  1  cut  ions  Systems  (COMM)  -  In  addition  to  the  normal  definition  of  this 
f :  1  "  -wing  definition  shall  apply  to  this  report:  systems  that 

.•  -  V -lew  SCO  vHz. 


Com  nun i cat  ions  Intelligence  (COMINT)  -  Technical  and  intelligence  information 
f”  m  foreign  communications  by  other  than  the  Intended  recipients 

m of  the  Army,  1  983) . 


Development  Testing  (DT)  -  Testing  of  materiel  systems  conducted  by  the 
.  developer  using  the  principle  of  a  single,  integrated  development  test 
•  ■•.■■m  T.3t  rate  that  the  design  risks  have  been  minimized,  that  the 

. ~  development  process  is  complete,  and  that  the  system  will  meet 


o  estimate  the  systea’s  military  utility  when  it  is 


:■  vol opnent  testing  is  conducted  in  factory,  laboratory,  and 
>n vironmonts ,  (Department  of  the  Army,  1976). 


Direction  Finding  (DF)  -  A  procedure  for  obtaining  bearings  of  radio  frequency 
:  *  vrs  :  v  using  a  highly  directional  antenna  and  a  display  unit  on  an 
receiver  or  ancillary  equipment  (GSA,  1986). 


Electromagnetic  Environmental  Test  Facility  (EMETF)  -  A  facility  operated  by 

•  .  .  Ar^v  Electronic  Proving  Ground,  Fort  Huachuca,  Arizona,  with  capabili- 
-.i. :  r  rer forming  laboratory  and  field  measurements,  data  base  development, 

•  v ‘  «•,  aluate  the  EMC/EMV  of  U.S.  Army  C-E  systems  and  equipment. 


'  mi  '  Counter-Countermeasures  (ECCM)  -  That  division  of  electronic  warfare 

•  •.  ■'  :  ns  taken  to  ensure  friendly  use  of  the  electromagnetic  spectrum 
- -e  enemy' s  use  of  electronic  warfare  (GSA,  1986). 


Electronic  Countermeasures  (ECM)  -  That  division  of  electronic  warfare 
.•  v  i-.r  ions  taken  to  prevent  or  reduce  an  enemy's  effective  use  of  the 

‘  t :  a  spectrum  (GSA,  1986) . 


\  _%S  A*  '/  */  V; A  / . 


Electronic  Order  of  Battle  (EOB)  -  A  listing  of  all  the  electronic  radiating 
equipment  cf  a  military  force  giving  location,  type,  function,  and  other 
pertinent  data. 

Electronic  Warfare  (EW)  -  Military  action  involving  the  use  of  electromagnetic 
energy  to  determine,  exploit,  reduce,  or  prevent  hostile  use  of  the  electro¬ 
magnetic  spectrum  and  action  to  retain  its  effective  U3e  by  friendly  forces 

(GSA,  19861. 

Electronic  Warfare  and  Intelligence  (EWI)  -  Electronic  warfare  is  defined 
immediately  above;  electronics  intelligence  is  the  second  definition  below. 

Electronic  Warfare  Support  Measures  (ESN)  -  That  division  of  electronic  warfare 
involving  actions  taken  under  direct  control  of  an  operational  commander  to 
search  for,  intercept,  identify,  and  locate  sources  of  radiated  electromagnetic 
energy  for  the  purpose  cf  immediate  threat  recognition.  Thus,  electronic 
warfare  support  measures  provide  a  source  of  information  required  for  immediate 
decisions  involving  electronic  countermeasures  (ECM),  electronic 
■.•■'■inter- countermeasures  (ECCM),  avoidance,  targeting,  and  other  tactical 
lc/m*-»r:t  of  forces.  Electronic  warfare  support  measures  data  can  be  used  to 
pr :  signal  intelligence  (SIGINT),  both  communications  intelligence  (COMINT) 

mi  electronics  intelligence  (ELINT)  (GSA,  1986). 

Electronics  Intelligence  (ELINT)  -  Technical  and  intelligence  information 
.ed  from  foreign  noncommunications  electromagnetic  radiations  emanating 
frmr  ■•.  vr  "nan  nuclear  detonations  or  radioactive  sources  (GSA,  1986). 

Intercept  -  '.  To  gain  possession  of  communications  intended  for  others  without 
•'■■;:•  ."■.'-•r.',  and,  ordinarily,  without  delaying  or  preventing  the  trans- 

•■'.  ssi  m;  .  Acquisition  of  a  transmitted  signal  with  the  intent  of  delaying  or 
■:  i tin .•*  receipt  of  that  signal  by  the  intended  user  (GSA,  1986). 

Measures  of  Functional  Performance  (HDFP’s)  -  The  set  of  bounds  or  parameters 
within  which  3  system  is  expected  to  normally  operate.  A  measure  of 

performance  is  an  essential  element  of  a  test  criterion. 

Non-Communications  Systems  (Non-COMM)  -  In  addition  to  the  normal  definition  of 
'his  word  (e.g.,  -adar,  navigation  aids),  the  following  definition  shall  apply 
a  this  report:  systems  that  operate  above  500  MHz. 

Reliability  Function,  R(t)  -  The  probability  that  a  system  will  operate 

trr :  jghout  'he  total  mission  time. 

Signals  Intelligence  (SIGINT)  -  1.  A  category  of  intelligence  information 
.<■;-■•  wi  communications  intelligence  (COMINT),  electronics  intelligence 

,  -n  ;  telemetry  intelligence;  2.  Intelligence  information  comprising, 
«••■  ;•  ...i-ally  ~r  in  combination,  all  communications  intelligence 

no  nits  intelligence  (ELINT),  and  foreign  instrumentation  signals 
.  .  g-'-o  ,  '  ,.';v«r  transmitted  (GSA,  1986). 

Stress  Loading  Facility  (SLF)  -  An  (envisioned)  integrated  and  automated  test 
•  *  -  th  will  '  -'arable  of  generating  a  dense  electromagnetic  threat  test 

■-■  *  ,  -  --.fa icing  bo'  h  COMM  and  Non-COMM  systems  and  equipments,  and 
.  '  •■  ...■-  trr;  eg  performance  parameters  of  a  SUT. 


1  lh 
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••’ican  National  Standard  for  inform, at  ion  systems— data 
’  services — user-oriented  performance  parameters"  on 
’S3) .  The  purpose  of  the  standard  is  "to  ■staMish  a 
tS,  assessing,  and  comparing  the  performance  of  data 
•  :  services  from  the  point  of  view  of  me  data 
mis  standard  subsequently  was  adopted  as  Federal 
•  A 5  '  ISA ,  \  935 ' . 

,  n  t  i<  “t  in  Ttar.dird  X3.102  is  the  Torus  an  user 
-  -’"nnoe  rather  than  on  engineering  design  consider,*- 
it  is  preside  to  describe  tnese  use-’-or  iented 
it  ire  system  independent;  i.e. ,  the  p  »r  ureters  may 
:  -•ommur.  i  •  «t  :  an  system  or  service,  irrospe ct  i  v<  of 

t >1 'vv,  -r  m.trol  protocol. 

*  •  sr  »  *  .  '  >  •*'»•}"] > j  j  ->  nj  *  i  on  3  V  3t  '  ’.ml.'4  t 

:  ;  r  -  it  •>  i  in  Figure  3-1  : 


_  a  t  : .  n  Process  Mode  j )  .  T n o  digital 

e-  d  i.-e-c  m ;  at  be  defined  using  a  user-.:'"  iented 
•  ‘  mn  r  •■  •••.*  .  This  definition  must  tdedify  tr;e 

*  w-'—r.  the  .sen's  arid  the  system,  the  information 

m-m  sy?‘em  interface,  ar.d  how  the  informal  ion 
io»r  s  y  s  t  **•  m  interface.  All  user /system 
.  •  jn  :  -r  .i  on  processes — may  be  defined  as 

:.-un*  ‘>d,  timed,  o-  compared.  Key  interface 

•  -.••.•>•■;  ....  cv t  -  t.e  used  to  calculate  perf  ‘irri'inee 
_y,  tnc re  r*'f‘*r  nee  events  specify  ail  informal  •  on 

i  comprehensive,  user-oriented  wav.  It 
"  m ay  *•«  -■ i t h.e-  ,  human  or  a  computer  application 
'■mnu  r  m  a  t  e  d  inf  jrmat  icn.  For  a  human  user,  the 

vid s  t  •,  the  physical  interface  between  the  human 
inal.  For  a  -oop  i*‘>r-ippl  i  cation-program  user,  the 
functional  interface  between  the  application 
-  v  stem.  Either  a  ; human)  terminal  operator  or  an 
data  rcec-din?  media  in  transferring  information  to 
from  n  sys*.  cm.  Typical  media  used  by  terminal 
.,  magnetic  stripe  cards,  punched  paper  tape,  and 
Typical  medi  i  used  by  application  programs  are 
d  isms.  Tr.  either  case,  these  data  media  arc 
■  n-r  tuar,  with  the  system.  Information  can  he 
int-mf  i r.  a  variety  of  ways.  Tynmal 
•  i  T''.te"f  me  a"«  manual  key  strokes  on  the 
:  -  iht  L  r.*  m  i  iso  living  of  received  •.  :  ‘  •  " . 

i  i  '  i‘  :  •  r  r  •  :  •  "at  i  nr  system  iri'orfuc'>  are 


? a t  i o r,  crogram  d a t a . 


;  y  o  l  c  a  i 


k 


-mv-  e  - ■  ■'VV'. m- -  •  /.f.  /m  O-'. 

*  ^  j«  '  •  *  ».  *  ■  '  »  *  »  m  *  -  ~  •  »  j*  *  k  m  *>  *  «  *  *  a.  *  »  v  »  *•»'»•  .  »  *  »  ■  ■  *  ■ 
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•  OPERATOR 

•  PROGRAM 


Function 

Definition 


Outcome 

Definition 


Parameter 

Selection 


COMMUNICATION 
User  Info  I  SYSTEM  I  *  User  ,nfo 
Overhead - •  Overhead 


Start  (Typical  Function) 
PERFORMANCE  TRIAL 

Possible  Discrete  Outcomes 


END  USER 

.  OPERATOR 
•  PROGRAM 


SUCCESSFUL 

PERFORMANCE 

(SPEED) 


INCORRECT 

PERFORMANCE 

(ACCURACY) 


NON¬ 

PERFORMANCE 

(RELIABILITY) 


FUNCTION 


ACCESS 


USER 

INFORMATION 

TRANSFER 


DISENGAGEMENT 


1  PERFORMANCE  CRITERION 

SPEED 

ACCURACY 

RELIABILITY 

<  SPECIFIC  Xf 
PERFORMANCE  \ 
PARAMETERS:  ' 

•  PRIMARY  j 
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snot  ion  Deficit ion.  Any  description  of  performance  must  relate  to  some 
notion's)  that  tne  system  is  expected  to  perform.  The  second  step,  then,  in 
fining  performance  parameters  (and  MOFPs)  for  digital  communication  systems 
th-'  definition  of  a  set  of  specific  communication  functions.  The  Standard 
f ir.es  three  primary  functions  (in  terms  of  reference  events)  as  follows: 

T:.~  ic cess  function  describes  a  user’s  "access  request."  An  access 
re  -it  begins  when  any  signal  is  applied  to  the  user/system  inter¬ 
face  for  tne  purpose  of  initiating  a  digital  communication  session 
ire  ends  when  the  first  bit  of  user  information  is  input  to  the 
ay  -.tern.  The  access  request  includes  dialing,  switching,  and  ringing 
.  ’  .  ’’.illy  associated  with  establishing  physical  circuits  and 
protocol  level  activities  such  as  are  associated  with 
dc.  vur.g  X . 25  virtual  circuits. 


na  t  ion  transfer  function  describes  user  exchanges  of 
*.  through  tne  system.  In  general,  information  transfer 
access  is  completed  and  ends  when  the  last  disengagement 
issues.  Information  transfer  includes  all  formatting, 
>n,  storage,  error  control,  and  media  conversion  activities 
iet wee r.  sta^t  of  transfer  and  completion  of  delivery.  Two 
l formation  transfer  functions  are  the  bit  transfer  function 
c*:  transfer  function.  The  bit  transfer  function  provides 
nisi  3  for  comparing  systems/services  that  use  different 
'tic  bloc--  lengths;  the  block  transfer  function  describes 
;  relative  to  an  information  unit  that  is  more  relevant  to 


gemor.t  function  describes  the  user's  "disengagement 
The  disengagement  request  begins  when  any  signal  is 


te  a  iser .'system  interface  for  the  purpose  of  terminating  a 
larticlrat  i  on  in  a  digital  communication  session  and  ends,  for 
?r,  when  1  i sengagement  has  been  requested  for  that  user  and 
ir  is  able  to  initiate  a  new  access  request.  Disengagement 
:  physical  circuit  disconnection,  where  required,  and  higher- 
"otocol  termination  activities  such  as  X.25  virtual  circuit 
as  appropr i ate. 


•isti-*  of  these  primary  communication  functions  is  that 
This  characteristic  means  that  successful  performance 
on  events  that  are  user-controlled.  Using  these 
r-'un ired  system  performance  creates  a  problem  in  that 
r  verier  has  m  control  over  user  practices  in  using  the 
is  a ver come  by  explicitly  describing  the  influence  of 
Ti-v  parameter  values  by  defining  separate  "ancillary" 
■  .SS-- !  lute-  in  this  Appendix. 


,.n ir  1  step  in  defining  performance  parameters  hand 
cation  systems  is  to  define  a  set  of  possible 
■  ir.a-v  parameters.  Three  general  categories  of 


.  r  o  n  era. 


1 1  Performance.  The  function  is  completed  within  a  specified 
maximum  performance  time,  and  the  result  or  output  is  exactly  as 
intended .  A  familiar  example  is  successful  connection  to  the  correct 
called  party  in  a  voice  telephone  call. 


Incorrect  Per  formano e .  The  funct  ion  is  completed  within  the 
specified  maximum  performance  time,  but  the  result  or  output  is  not 
as  intended.  A  familiar  example  is  connection  to  a  wrong  number  in  a 
v-'i  'e  telephone  call  as  a  result  of  a  system  switching  error). 


Var.perf  crr.a.nce .  The  function  is  not 
maxima-:  ;  er formance  time.  A  familiar 
i  1  ephone  call  attempt  by  the 


completed  within  a  specified 
example  is  the  blocking  of  a 
system  (as  indicated  by  a 


. *  •  rm.e.n  are  significant  because  they  correspond  with  three  basic 
'oncerns  of  digital  communication  users.  Successful  performance  is 
-  with  a  user's  concern  for  speed  (delay  or  rate),  incorrect  perform- 
i  •  >*c:  with  i  user's  concern  for  accuracy,  and  nonperformance  is 

i  us*”'.'  concern  fc”  reliability.  These  general  performance 

*.  as  a  framework  for  organizing  the  primary  parameters.  The 
-m  tp  ■  >'  and  nonper formance  outcome  categories  are  divided, 
t  m.-c  detailed  outcomes  and  a  more  comprehensive  outcome 

;  i  .  Tn*-1  more  detailed  possible  outcomes  of  a  primary  function,  for 

; -r formance  trial,  are: 


?»?rf  ;rmance.  The  expected  result /output  occurs  and  is 

location  and  content. 


-irr  or.  7  no  expected  result /output  occurs  at  the  correct 

-  :*  is  in-.'””-^t  in  content. 


The  expected  result /output  occurs  at  an  incorrect 


.  ■:  t ” .i  avert .  An  unexpected  result /output  occurs  in  addition  to  that 

.a  ^ 


:  Vo nper formance .  The  expected  result/output  does  not  occur 

•re  maximum  performance  time  either  as  a  result  of  the  system 
.m  a  slocking  'busy)  signal  or  due  to  excessive  delay  by  the 

I*  :■  nr.  er  forma r.ce .  The  expected  result/output  does  not  occur 

tne  maximum  performance  time  either  as  a  result  of  the  user 
:  r.  1  m  king  V- . sy':  signal  or  due  to  excessive  delay  by  the 


•  V  : or.es  defined  by  the  Standard  for  each  of  the  primary  functions 

*  he  •'■.‘.mom-  sample  space  matrix  shown  in  Figure  B-2.  Note  that 

:  r”t  mu-  -  sense  and  are  not  defined  for  the  access  and 


l 


c. 
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arameter  Selection.  The  final  step  in  defining  performance  parameters 
1 F P3  for  digital  communication  systems  is  to  select  and  define  a  minimum 
r  parameters  to  describe  performance  relative  to  each  function  and 
As  noted  earlier,  this  process  results  in  primary  (user  dependent) 
ters  and  ancillary  parameters  that  express  the  user's  contr ibution(s)  to 
ed  delays.  In  performing  this  step,  the  Standard  defines  21  parameters 
c.n  i"  are  primary  parameters  and  4  are  ancillary  parameters.  Each  of 
parameters  is  defined  in  mathematical  form  in  the  Standard  (ANSI,  1983) 
'.’s er  Reference  Manual  (Seitz  and  Grubb,  1983)  for  the  Standard.  Of  the 
"v  parameters,  ^  relate  to  the  access  function,  11  relate  to  the 
■ilior.  transfer  function,  and  2  relate  to  the  disengagement  function.  The 
ters  ire  listed  below  and  summarized  in  Figure  B-3,  organized  by  function 
••'form.ince  criterion  and  by  function  and  performance  parameter  type  for 

Pr imary  Parameters 


A:  teas  Time 

'.c  orrect  Access  Probability 
•■  •ess  Denial  Probability 
A  -ess  outage  Probability 
■'■it  Error  Probability 
Pit  Misdelivery  Probability 
Extra  Bit  Probability 
■:t  Toss  Probability 
•loo*  Transfer  Time 
Bloc*  Error  Probability 
PI  '0*  Misdelivery  Probability 
Extra  Block  Probability 
Block  boss  Probability 
:S‘.‘r  Information  Bit  Transfer  Rate 
Transfer  Denial  Probability 
Disengagement  Time 
Disengagement  Denial  Probability 


Ancillary  Parameters 

■ser  Fraction  of  Access  Time 
is er  Fraction  of  Block  Transfer  Time 
iser  Fraction  of  Input/Output  Time 
is-r  Fraction  of  Disengagement  Time 

:.i^nal  secondary  (or  availability)  parameters,  so  termed  to  emphasize 
o.  it  they  are  defined  on  the  basis  of  measured  primary  parameter 
v?r  than  on  the  basis  of  direct  observations  of  interface  events, 
•.  »ly  related  to  the  Standard  have  been  defined  in  a  paper  by  Seitz 
’)3C;.  These  parameters  provide  a  macroscopic,  long-term 
;  iesc*’ iption  in  terms  traditionally  associated  with  the  concept  of 
:y.  These  secondary  performance  parameters  are: 


ne  Between  -Outages.  The  average  value  of  elapsed  time 
r.  taring  and  next  leaving  the  Operational  Service  state 
known  as  the  mean  time  between  failure,  MTBF). 


FUNCTION 


SPEED 


PERFORMANCE  CRITERION 
ACCURACY 


RELIABILITY 


PERFORMANCE 

TIME 

ALLOCATION 


■ - 

ACCESS  DENIAL 

ACCESS 

access  time 

INCORRECT  ACCESS 
PROBABILITY 

PROBABILITY 

ACCESS  OUTAGE 

PROBABILITY 

BIT  ERROR  PROBABILITY 

BIT  MISDELIVERY  PROBABILITY 

BIT  LOSS 
PROBABILITY 

BLOCK  TRANSFER 

EXTRA  BIT  PROBABILITY 

USER 

INFORMATION 

TRANSFER 

TIME 

BLOCK  ERROR  PROBABILITY 

BLOCK  MISDELIVERY  PROBABILITY 

EXTRA  BLOCK  PROBABILITY 

BLOCK  LOSS 
PROBABILITY 

USER  INFORMATION 

f 

i 

BiT  TRANSFER 

RATE 

TRANSFER  DENIAL  PROBABILITY 

! 

!.  'ENGAGEMENT 

0  -ENGAGEMENT 
TIME 

DISENGAGEMENT  DENIAL  PROBABILITY 

•  by  (unction  and  performance  criterion. 


Legend 


□ 

EZ3 


Primary  Parameters 
Ancillary  Paramelers 


f - 

! 

:  UNCTION 

■ 

PERFORMANCE  PARAMETER  TYPE 

DELAY 

(IF  COMPLETED) 

RATE 

(IF  COMPLETED) 

FAILURE 

PROBABILITY 

ACCESS 

»  ACCESS  Time 

•  ...::es  fraction  of 

ACCESS  ''ME 

•  INCORRECT  ACCESS 

•  ACCESS  OUTAGE 

•  ACCESS  DENIAL 

USER 

INFORMATION 

TP.AV>FFS 

•  BLOCK  TRANSFER  TIME 

•  USER  FRACT  ON  OF 

BLOCK  TRANSFER  TIME 

•  USER  ERAST. ON  OF 
'Wr OUTPUT  TiME 

•  USER  INFORMATION 

BIT  TRANSFER  RATE 

•  BIT  ERROR 

•  BIT  MISDELIVERY 

•  EXTRA  BIT 

•  B:T  LOSS 

•  BLOCK  ERROR 

•  block  Misdelivery 

•  EXTRA  BtOCK 

•  BLOCK  toss 

•  TRANSFtR  DENIAL 

■  v  .  v;-n: 

•  DISENGAGEMENT  T'ME 

•  USER  FRACTION  OF 

L  '  i  N .j  * ' ; T M L *J T  TiME 

... 

_ 

•  DISENGAGEMENT 

DENIAL 

OrQjtnszatio’i  hy  function  and  performance  parameter  type. 


i  j  i  ;  1 1 . »  •,  1  j'  it;  u'--‘  -  ■  a*.1  . 


< .*■*  I  : 


Jut age  Duration.  The  average  value  of  elapsed  time  between  entering 
and  next  leaving  the  Outage  state  (sometimes  known  as  the  mean  time 
to  repair ,  MTTR) . 

Outage  Probab i lity .  The  ratio  of  total  message  transfer  attempts 
resulting  in  the  Outage  state  to  total  message  transfer  attempts 
included  in  the  measurement  sample. 

These  secondary  parameters  as  used  to  define  the  concepts  of  availability  and 
unavailability  are  illustrated  in  Figure  B-4. 

‘■'unh  additional  definition  and  explanation  of  the  Standard  are  given  in  the 
references  that  have  been  cited  and  material  referenced  in  those  documents. 
One  additional  source  of  considerable  use  in  applying  the  Standard  to  system 
performance  measurements  is  a  report  that  defines  and  describes  measurement 
m-'thods  for'  user-oriented  performance  evaluation  (ANSI,  1986). 


>  s 
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•AND ED  OUTLINE  OF  A  DETAILED  TEST  PLAN  FOR  (TYPE/PHASE)  TEST  OF 
iMENCLATURE  OF  TEST  ITEM) 


with  the  use  of  subsections  as  appropriate,  will  present 
ne  development  of  the  equipment/system,  a  brief  description  of 
system,  clear  statement! s)  of  test  objective! s) ,  and  a 
.he  scope  of  the  test. 


i v.  the  .-•••  of  subsections,  will  define  and  describe  the  tests 
...  'or.du  'ted,  using  the  SLF  to  create  the  desired  test 
30  as  t  •>  •  reduce  and  collect  the  interface  events  that  must  be 
counted  reduction  and  analysis  (as  described  in  Section  6) 
••for  it  an  oo  in  accordance  with  the  user-oriented  functions  and 
a  motors  th-.t  have  been  selected  for  use  in  the  test.  Such 
aot-’r  i  •  ed  as  system  independent. 


ire  to  determine  that  the  system  to  be  tested  is  complete  and  in 
•  ,,t  (Condition  and  that  any  required  support  equipment  is 
.  r.  :  »n  normal  operating  condition  prior  to  start  of  the 


Appropri  at-'  Regulation) 

.  ..  tested  shall  be  complete  and  in  normal  operating 
o  : .  :r’L:'  to  start  of  the  test(s). 

;:-.ort.  e  i  iipmeht  required  for  trie  system  to  be  tested  shall 
ladle,  'omclete,  and  in  normal  operating  condition  prior 

•t  Of  th*‘-  t--St(s). 


;  s  :r--n  v:  •  ies  existent  for  the  system  to  be  tested 

me  ies  existent  for  required  support  equipment 
•  ■■-;  *•••:  y  >  “<--sted 

•:  ntat  i  on  of  any  physical  damage  existent  for 

?t.;d  and/or  the  support  equipment  required  for 


r,e  n  t  s 


nd  ren  a 


performed  and  all 


i 


i.1.4  Data  Acquisition  Procedure 


The  test  officer  and  other  technical  and  maintenance  personnel,  as  required, 
will  conduct  the  pretest  system  check-out.  This  check-out  will  include: 


a.  Unpack  and  inventory  the  system  to  be  tested  and  all  required 
support  equipment  and  compare  the  contents  with  packing  list  to 
determine  if  any  discrepancies  exist. 


b.  Inspect  the  system  to  be  tested  and  all  required  support 
equipment  evidence  of  physical  damage. 


Perform  a  pretest  operational  performance  verification  check  on 
the  system  to  be  tested  and  all  required  support  equipment,  as 

appropriate. 


Adjust  and/or  repair  each  discrepant  condition,  if  possible. 


Record  all  discrepant  conditions,  including  those  removed  by 
adjustment  and/or  repair  as  well  as  those  not  removed. 


\r.alvt ical  Procedure 


Tr.v  lata  will  be  used  to  assist  the  test  officer  in  determining  if  the  system 
to  be  tested  and  all  required  support  equipment  are  complete,  undamaged,  in 


'rating  condition,  and  ready  for  testing. 


detection  Function 


1 . . 1  Objective! s) 


Tro  objective! s)  of  detection  function  testing  must  be  defined  in  a  3hort, 

clear  statement. 


2.2.2  Criteria  (Appropriate  Regulation) 


The  criteria  for  successful  detection  function  outcomes  must  be  defined  along 
with  the  basis  for  the  criteria.  At  least  two  criteria  are  applicable  to  each 
test;  these  criteria  are  the  time  for  successful  detection  for  each  trial  which 
is  the  basis  for  calculating  the  detection  rate,  and  the  percentage  of 
successful  (trial)  detections  that  are  required  for  a  successful  performance 


5  Data  Required 


a.  V:«>  start  time  for  each  detection  attempt 


'.he  function  stoo  time  for  each  successful  detection 


e.  tne  tatal  number  of  detection  opportun ities  (trials) 


the  total  number  of  successful  detections 


f*.4 w- 

/ 


O/,  -\jr.  r.  W  - 


lumber  'f  incorrect  detections 


number  of  nondetections. 

:  ior.  Procedure 

quired  using  the  SLF  and  interface  monitors  as  described  in 

rocedur^s 


educe"  and  analyzed  as  described  in  Sections  6.3  and  6.H. 
s  that  will  be  calculated  include: 

tim*‘  'fir  each  trial) 

't-'ctiin  time  (calculated  for  each  performance  period 
rich  factor  combination  of  the  test) 

tact5  an  probability  (calculated  for  each  factor 

r.  of  the  test) 


on  pr  :  ability  (calculated  for  each  factor  combination 

r 

cri.iati.on  Function 


3irn  si  characterization  function  testing  mu3t  be  defined  in 


r-"opr  i  U-e  Pcgulat  ion) 


i  v.'ossfui  signal  characterization  function  outcomes  must  be 
the  basis  for  the  criteria.  Several  criteria  that  apply 

or  successful  signal  characterization  for  each  trial 

ibV‘  t  ‘-lerance  in  measuring  signal  frequency  (may  be 
or  W  carrier) 

t  -  nee  in  measuring  PW,  if  carrier  is  pulsed 
•  -  f  -'ranee  in  measuring  PRF/PRI,  if  carrier  is 

0  successful  signal  characterizations  that  are 
ir  o  •-.uc-essful  performance  period. 


a.  tne  start  time  for  each  signal  characterization  attempt 

b.  the  function  stop  time  for  each  successful  signal 
characterization 

c.  tne  total  number  of  signal  characterization  opportunities 
(usually  not  the  total  number  of  trials,  because  signal 
characterizations  are  attempted  only  when  the  detection  function 
outcome  has  been  successful) 

the  total  number  of  successful  signal  characterizations 

e.  trie  total  number  of  incorrect  signal  character izations 

f.  tne  total  number  of  signal  noncharacter izations. 

lata  Acquisition  Procedure 

data  will  be  acquired  using  the  SLF  and  interface  monitors  as  described  in 

ion  6.2. 

Analytical  Procedures 

i*ta  will  be  reduced  and  analyzed  as  described  in  Sections  6.3  and  6.1!. 
parameter  values  that  will  be  calculated  include: 

a.  signal  characterization  time  (for  each  trial) 

.  average  signal  character ization  time  (this  parameter  may  be 
calculated  for  each  performance  period  and/or  for  each  factor 

combination  of  the  test) 

c.  incorrect  signal  characterization  probability  (this  parameter 
would  be  calculated  for  each  factor  combination  of  the  test) 

i.  signal  noncharacter ization  probability  (this  parameter  would  be 
calculated  for  each  factor  combination  of  the  test). 

Pm  i  1 1  c  Identification  and  Location  (EIL)  Function 

1  Object i ve( s) 

objective's'  of  El;,  function  testing  must  be  defined  in  a  short,  clear 


•c i a  Appropriate  Regulation) 

"  successful  EIL  function  outcomes  must  be  defined  along  with 
•’  tne  criteria.  Several  criteria  that  apply  include: 


■  im.e  for  successful  emitter  identification  and  location  for 
trial 


liveable  tolerance  in  measuring  a  line  of  bearing 

allowable  tolerance  in  elliptical  error  probability  when 
lit ing  a  position  from  two  or  more  lines  of  bearing  that 
teen  measured  from  different  locations 

er tentage  of  successful  EIL  trials  that  are  required  for  a 
iful  performance  period. 

i  m»y  be  the  function  of  primary  interest,  since  successful 
this  function  is  dependent  upon  successful  completion  of  the 


*  tin--  f -r  each  EIL  attempt 

.net:  >n  stop  time  for  each  successful  EIL 

.til  r.jri'-t)"  of  successful  opportunities  (usually  not  the 
number  of  trials  nor  the  total  number  of  successful  signal 
.  'ter  i.’.at  i  ins ,  because  the  EIL  function  is  attempted  only 
the  signal  -haracter izat ion  function  outcome  has  been 


■til  number  of  successful  EIL  functions 
:tnl  -limber  if  incorrect  EIL  functions 
etui  numb-r”  -f  EIL  function  nonperformances. 

■;u:  sit  ion  Procedure 

bo  acquired  using  the  SLF  and  interface  monitors  as  described  in 


-  a  1  Procedures 

be  reduced  and  analyzed  as  described  in  Sections  6.3  and  6.H. 
values  that  will  be  calculated  include: 

identification  and  location  time  (for  each  trial) 

EIL  "  im.r  'this  parameter  may  be  calculated  for  each 
■'r, nor  iod  and/or  for  each  factor  combination  of  the 


EIL  nro> -ability  (this  parameter  would  be  calculated  for 
-‘omninat  ion  of  the  test) 


SSSwS's'! 


EIL  r.onper  Torrance  probability  (this  parameter  would  be 
calculated  for  each  factor  combination  of  the  test). 


.1 . 5  System  f'perab  1 1  i t y  State  Function  (Secondary) 

?  .  5  . 1  Object ive( s) 


Toe  object ive, s'  for  the  system  operability  function  must  be  defined  in  a 

short,  clear  statement. 

.1 .  -  .  Cr  i  ter  1  i  '  App-opr i ate  Regulat  Ion) 

The  or  1  »  f  r  the  system  to  be  in  an  "Available"  (operational)  state  and  the 

t  ?  dp  tar  !•*  f^e^uency  and  diration  of  "Unavailable"  (nonoperational)  states  must 

■  define!  llcr.g  with  th®  basis  for  the  criteria: 

i.  required  successful  function  outcomes  in  order  for  the  trial 

c  .vr.e  to  be  successful 

■  <  percent  ige  -f  successful  trial  outcomes  that  are  required  for 

i  successful  performance  period 

tne  fraction  cf  total  test  time  that  the  system  must  be  in  an 

operational  state  (Available) 

:.  the  requirement  for  minimum  time  that  the  system  must  be  in  an 

rpenhing  state  between  failures 

tr.e  -equirement  for  maximum  time  that  the  system  may  be  in  a 

nonoperating  state  (failed). 

*  :  at  a  Required 


All  as'- a  required  for  the  (secondary)  system  operability  state  function  are 
:ordd  lata  required  for  the  primary  functions.  Those  data  are: 

a.  the  start  time  for  the  test 

b.  the  time  that  each  performance  period  starts  (if  different  from 
the  test  start  time  for  the  first  performance  period  or  the  end 
:f  t.hrt  preceding  performance  period  for  subsequent  performance 

per  ids) 

the  time  that  each  performance  period  ends 

d.  t time  that  the  test  ends  (if  different  from  the  end  of  the 

L-iSt  performance  period  of  the  test) 


de  outcome  for  each  function  attempted 

the  outcome  for  each  trial 

"h»'.  utoomo  for  «ach  performance  period 
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the  total  number  of  trials  during  each  performance  period 


the  total  number  of  successful  trials  during  each  performance 
period 

the  total  number  of  performance  periods  during  the  test 

the  total  number  of  successful  performance  periods  during  the 

t  es  t . 

t a  Acquisition  Procedure 

-ita  will  be  acquired  to  describe  the  primary  function  outcomes  using 
an  1  interface  monitors  as  described  in  Section  6.2.  Additional 
ions  using  the  primary  data  and  the  results  of  primary  function 
are  required  to  letermine  the  system  operability  state. 

1 1 y 1 1 c  a  1  Procedures 

will  no  reduced  ml  analyzed  as  described  in  Sections  6.3  and  6. *4.  As 
ve,  tv-  data  used  to  calculate  the  secondary  parameters  are  primary- 
•  t  it  *.  -and  primary-function  outcome  data.  The  secondary-parameter 

at  will  be  calculated  include: 

v  a  t- 1 1  *■_.  -a  3 1,  tin  f ’ 


the  elapsed  time  far  each  successful  performance  period 

the  aggregate  elapsed  time  for  all  successful  performance  periods 

tv-  •■'lapsed  time  for  each  failed  performance  period 

tv-  segregate  elapsed  time  for  failed  performance  periods 

iva liability,  as  the  ratio  of  aggregate  successful  performance 

tine  to  total  test  time 


the  average  system  operating  time  between  failures 

the  average  system  failure  time. 

wF  INS  CRUM EM  TED  WORKSHOP  TESTS  (BENCH  TESTS) 

ton,  with  the  use  of  subsections,  will  define  and  describe  the  bench 
r,  will  be  required  to  verify  most  system  and  component  specifications 
.-■■on  used  to  define  required  performance  in  terms  of  system  specific 
v ■  r  ir.f- orient---  i)  parameters.  Bench  tests  to  determine  values  for 
:  1  v  *«r  i  ng-  or  lent  e.i  parameters  often  will  help  the  test  officer 
:  son--  -'f  t.v  user-oriented  (system  independent)  performance  results, 
illy,  if  will  be  necessary  to  perform  bench  tests  to  obtain 
i  o i «nt e  :  performance  data  that  are  required  to  describe  system 
wh—j  c'-mr-ute’'  simulations  are  to  be  conducted.  These  bench  tests 
.  ,  -*  ;f  i  v-*'  in.-  of  tests  that  begin  with  the  Pretest  System, 


%  > 


■Subsystem,  or  Component  Check-out  and  span  all  the  tests  required  to  be 
performed,  numbered  3.1  through  3.N  (where  N  =  the  maximum  number  required). 
Very  often  these  bench  tests  will  include  Receiver  Characteristics  Tests 
''3.25,  Antenna  Character istics  Tests  (3.3),  and  System  Characteristics  Tests 
53.4),  but  other  tests  may  be  identified  as  required. 

*.l  Pretest  System,  Subsystem,  or  Component  Check-out 

3.1.*  2b j  ect i ve( s) 

The  objectives  are  to  determine  that  the  system,  subsystem,  or  component  to  be 
t os  tel  is  complete  and  in  normal  operating  condition  and  that  any  required 
support  equipment  is  available,  complete,  and  in  normal  operating  condition 

prior  tA  start  of  the  test(s). 


Cr i ter  ia 


Regulation) 


a.  The  system,  subsystem,  or  component  to  be  tested  shall  be 
complete  and  in  normal  operating  condition  prior  to  start  of  the 

test( si . 

b.  Any  support  equipment  required  for  the  system,  subsystem,  or 
component  to  be  tested  shall  be  available,  complete,  and  in 
normal  operating  condition  prior  to  start  of  the  tes’s). 

3  Data  Required 


record  of  discrepanc ies  existent  for  the  system,  subsystem,  or 
component  to  be  tested 

’•veord  of  discrepancies  existent  for  required  support  equipment 
for  the  system,  subsystem,  or  component  to  be  tested 

photographic  documentation  of  any  physical  damage  existent  for 
the  system,  subsystem,  or  component  to  be  tested  and/or  the 
support  equipment  required  for  the  system,  subsystem,  or 

component  to  be  tested 

record  of  all  pretest  adjustments  and  repairs  performed  and  all 

performance  checks  not  met. 

it. a  Acquisition  Procedure 


Tee  test  officer  and  other  technical  and  maintenance  personnel,  as  required, 
will  conduct  the  pretest  system,  subsystem,  or  component  check-out.  This 

'■’.vck-  jut  wilt  include: 

a.  Unpack  and  inventory  the  system,  subsystem,  or  component  to  be 
tested  and  all  required  support  equipment  and  compare  the 
contents  witn  packing  list  to  determine  if  any  discrepancies 


Inspect  the  system,  subsystem,  or  component  to  be  tested  and  all 
required  support  equipment  for  evidence  of  physical  damage. 

Perform  a  pretest  operational  performance  verification  check  on 
the  system,  subsystem,  or  component  to  be  tested  and  all  required 
sup; ort  equipment,  as  appropriate. 


ijust  and/or  repair  each  discrepant  condition,  if  possible. 


e.  Record  all  discrepant  conditions,  including  those  removed  by 
i  ijustment  and/or  repair  as  well  as  those  not  removed. 

Ana  ly tical  Procedure 

:nta  will  be  used  to  assist  the  test  officer  in  determining  if  the  system, 
ay  stem,  or  component  to  be  tested  and  all  required  support  equipment  are 
:  1  r t ,  undamaged,  in  operating  condition,  and  ready  for  testing. 

•Receiver  Characteristics  Tests 


'  ;ect  ive(  s) 

of  receiver  characteristics  testing  must  be  defined  in  one  or 
■  short,  clear  statement^ s) .  Different  objectives  may  be  necessary  for 

'erect  characteristics  tests  of  the  receiver. 


>i ter  ia  ' Appropr iate  Regulation) 

terion  for  performance  of  the  receiver(s)  relative  to  each  of  the 
;ter 1st ies  that  Is  tested  must  be  defined  along  with  the  basis  for  each 


•  ■'  1  - 1 a  Require  1 

i  n  P.Iv  required  to  define  characteristics  of  a  receiver  are  defined  by 
-CTD-u-.qc  (Department  of  Defense,  1963  and  1965)  and  include  the  items 
ted  below,  as  applicable.  Items  that  are  important  to  the  characterization 
a  particular  electronic  surveillance  system  receiver  may  not  be  included  in 
i  standard  and  should  be  added,  such  as  items  k,  1,  and  m. 

i.  tne  types  of  signals  (modulations)  to  which  the  receiver  will 

respond 

sensitivity  as  a  function  of  frequency  for  each  of  the  types  of 
signals  to  which  the  receiver  is  intended  to  respond 

selectivity  as  a  function  of  frequency  for  each  of  the  types  of 
signals  to  which  the  receiver  is  intended  to  respond 

d.  spurious  responses 

<■:.  overall  susceptibility  at  spurious  response  frequencies 


f.  intermodulation  characteristics 

g.  pulse  desensitization 

h.  CW  desensitization  and  adjacent  signal  interference,  as 
appropr iate 

i.  dynamic  range 

j.  oscillator  radiation  test 

< .  signal  detection  bandwidth 

l.  sweep  time  for  frequency  tuning 

m.  resolution  and  accuracy  in  measuring  the  carrier  frequency  of  a 
received  signal  (for  each  type  of  signal  to  which  the  receiver  is 
intended  to  respond)  when  operated  in  the  discrete  tuning  and 
scanning  modes,  as  appropriate. 

3.2.4  Data  Acquisition  Procedure 

The  data  will  be  acquired  using  the  Instrumented  Workshop  in  accordance  with 
the  measurement  setups  and  procedures  outlined  in  MIL-STD-449C  (Department  of 

Defense,  1963  and  1965). 

3  ...  5  Analytical  Procedures 

The  data  will  be  reduced,  analyzed,  and  presented  in  formats  consistent  with 
the  instructions  for  presentation  of  the  data  given  in  MIL-STD-449C. 

3 . ?  Antenna  Subsystem  Characteristics  Tests 

3 .3.1  Objective! s) 

The  object! ve( s)  of  antenna  performance  testing  must  be  defined  in  one  or  more 
shjrt,  clear  statement s) .  Different  objectives  may  be  necessary  for  different 
characteristics  tests  of  the  antenna  subsystem. 

3.3.2  Cr i teria  ( Appropriate  Regulat ion) 

The  criteria  for  performance  of  the  antenna  subsystem  relative  to  each  of  the 
characteristics  that  is  tested  must  be  defined  along  with  the  basis  for  each 

criterion. 

3.3.3  Data  Required 

Data  normally  required  to  define  characteristics  of  a  surface-based  antenna  are 
defined  by  MIL-3TD-449C  (Department  of  Defense,  1963  and  1965).  Data  normally 
required  to  define  characteristics  of  an  airborne  antenna  are  defined  in 
MIL-A-371  36  (U.S.  Air  Force,  1979).  Measurements  of  the  properties  that 
characterize  antennas  also  are  comprehensively  defined  in  the  IEEE  Test 
Procedure  for  Antennas,  Humber  149  (1965).  Items  that  are  important  to  the 
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it  ion  o!  ar.  antenna  subsystem  for  a  particular  electronic 
system  may  net  be  included  in  these  standards  and  should  be  added, 
Items  that  normally  are  important  to  defining  the  characteristics 
include: 

ronmental  conditions  that  affect  antenna  performance 

lotion  patterns  as  a  function  of  space  coordinates  and 
viency  that  ”epresent  amplitude,  phase,  and  polarization 
ertie-s  of  tr>'  antenna 

power  -Vila  and  directivity  of  the  antenna  as  a  function 


a*  ier.  efficiency  of  the  antenna  as  a  function  of  frequency 

tnn  mutual  impedance  character ist ics  of  the  antenna  as  a 

tier,  of  frequency 

e  temperature  of  the  antenna  as  a  function  of  frequency. 

.c.;uisition  Procedure 

.  be  acquired  using  the  Instrumented  Workshop  and  the  antenna  test 
'ordanee  with  the  measurement  setups  and  procedures  outlined  in 
department  of  Defense,  1963  and  1965),  MIL.-A-871  36  (U.S.  Air 
and/or  the  IEEE  Test  Procedure  for  Antennas  (  1965  ),  as 


. i aal  Procedures 

1  be  reduced,  analyzed,  and  presented  in  formats  consistent  with 
t  ions  for  presentation  of  the  data  given  in  MI  L-STD-H4  9C , 
and /or  the  IEEE  Test  Procedure  for  Antennas,  as  appropriate. 

haracter ist ics  Tests 

ive(s) 

e(s)  of  any  system  performance  testing  that  may  be  required,  to 
w  and/or  Field  Facility  testing  and/or  Computer  Simulation,  must 
i  one  or  more  short,  clear  statement! s) .  Different  objectives  may 
for  different  character istics  tests  of  the  electronic  surveillance 
stem  character  istics  tests  will  r.ot  be  routinely  performed,  but 
y  ;o  required  to  understand  performance  that  cannot  be  understood 
,ioq  SLF  and/or  Field  Facility  testing  as  deemed  necessary  and/or 
I  at  ion . ; 

•  1  -a  (Appropriate  Regulation) 


a  for  performance  of  the  system  relative  to  each  of  the 
os  tr.at  is  selected  for  IWS  testing  must  be  defined  very  care- 
with  the  basis  for  each  criterion,  so  as  to  provide  sharp  focus  to 
:m  character ist ics  tests. 


V‘;/‘ 


3.4.3  Data  Required 


Data  required  to  defire  system  characteristics  ( engineer ing-oriented 
specif ications)  normally  will  be  system  specific,  i.e.,  different  sets  of  data 
for  different  systems.  It  will  not  be  normal  procedure  to  perform  system 
characteristics  tests  of  this  type,  but  such  tests  may  be  necessary  under 
unusual  situations  to  understand  performance  that  has  been  observed  for  the 
user — oriented  tests.  The  types  of  data  that  may  be  of  interest  for  electronic 
surveillance  systems  include  the  following: 

a.  system  accuracy  in  measuring  carrier  frequency  and  modulation 
characteristics,  e.g.,  pulse  width,  pulse  repetition  frequency, 
or  pulse  repetition  interval,  etc. 

b.  bearing  accuracy 

:.  elliptical  error  accuracy  { intersections  of  multiple  bearings 
from  multiple  locations 

d.  signal  processing  times 

e.  signal  sorting  capabilities 


f.  system  clock  accuracies 

g.  speed  in  recording  and /'or  printing  output  information 

h.  self-calibration  capabilities 

i.  self-test  capabilities 

j.  data  link  capabilities. 

Data  Acquisition  Procedure 

The  data  will  be  acquired  using  the  Instrumented  Workshop  and  other  "bench 


test"  testing  capabilities  as  may  be  required  and  appropriate.  Data  acquisi¬ 
tion  will  be  in  accordance  with  measurement  setups  and  procedures  outlined  in 
appropriate  and  applicable  standards  and  good  engineering  practices. 

- • “ • b  Analytical  Procedures 


a 
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The  data  will  be  reduced,  analyzed,  and  presented  in  formats  consistent  with 
instructions  for  presentation  of  the  data  given  in  appropriate  and 
applicable  standards  and  in  accordance  with  good  engineering  practices. 

.  DE TAILS  OF  COMPUTER  SIMULATION 

This  section,  with  the  use  of  subsections,  will  define  and  describe  computer 
simulation  that  may  be  required  to  estimate  system  performance  under  normal 
operating  conditions  and  conditions  of  intentional  and/or  unintentional 
interference.  Computer  simulation,  though  it  may  be  much  less  expensive  than 
"esting,  usually  will  require  extensive  effort  to  prepare  the  required  input 


a 

£ 


data. 


of 

Si; 

an 


These  data  must  describe  the  technical  and  operational  characteristics 
every  C-E  system  and  item  of  equipment  that  will  comprise  the  environment, 
h  data  to  describe  the  technical  characteristics  of  receivers,  transmitters, 
antennas  must  either  be  estimated  or  obtained  from  measurements  performed 
the  I  Vi  S  tench  tests).  There  also  must  be  criteria  and  data  for  evaluating 
performance  of  systems,  both  intended  performance  and  performance  in 
por.se  to  interfering  signals.  Data  to  describe  the  operational  characteris- 
s  of  the  environment  must  be  developed  by  military  scientists  through  a 
cess  that  can  become  very  tedious.  Furthermore,  computer  simulation  can 
'•'lie  only  "snapshots"  of  system  performance  in  a  modeling  of  the  operational 
i-."'.  scenario'  that  may  be  of  interest. 


r r  ■  * p  1 at  ion  of  I  nput  Data 


i  ves 


evti  ve-  s'.  to  fcc  met  in  preparing  the  input  data  for  computer  simulation 
is.  must  be  cleanly  and  concisely  stated.  Different  objectives  will  i>e 
ry  v.  satisfy  different  objectives  for  the  computer  simulation. 

Dr : ter  1  ;  ' Appr opr i ate  Regulat ion) 

tori  a  to  be  followed  in  preparing  the  input  data  for  computer  aim  .l  at {or, 
ie fined  along  with  the  basis  for  each  criterion. 


••.!.}  Data  Required 


u ;  red  to  perform  computer  simulation  normally  include  technical 
oter i/.ations  of  all  C-E  systems  and  equipments  to  be  included  In  the 
r  L  :  leployment  along  with  specification  of  the  operational  plan  that  is  to 
y.ei.  This  composite  of  information,  known  as  the  electronic  order  of 

■'  <  rl-.h.  ,  will  include: 

.  listing  of  all  radiating  and  receiving  (C-E)  equipment  of  the 
military  forces  that  are  represented  in  the  scenario 

t.  technical  character  ist  ics  (e.g.,  modulation,  radiated  power, 
emission  spectrum,  receiver  bandwidth,  receiver  sensitivity, 
unv-nrin  pattern,  etc.)  for  each  item  of  C-E  equipment  that  is 

r r  ouented  in  t.ne  scenario 

geometry  of  the  scenario  deployment  (location  of  each  item  of 

>:  cu  L orient ) 


g uti  oral  function  of  each  item  of  C-E  equipment,  i.e.,  all 
i : :  i  ro>-  of  t  -  a  mm  i  t  ter  /r  ece  i  ver  pairs  that  the  simulation  will 


at  i  A-.’qu is i t ion  Procedure 

innings  of  C-E  equipment  to  be  included  are  developed  from 

n.prop-’iate  Field  Manuals. 


b.  Technical  characteristics  for  C-E  equipment  to  be  included  are 
obtained  from  bench  measurements  or  estimated  (e.g.,  spectrum 
synthesis,  receiver  passband  synthesis,  antenna  pattern  synthesis 
and  maximum  gain  calculation,  etc.). 

c.  Scenario  deployments  and  the  linking  of  C-E  equipments  to 
simulate  communications  during  combat  situations  are  developed  by 
military  scientists  who  study  present  and  future  concepts  to 
develop  data  that  define  the  deployment  of  the  military  units  in 
expected  combat  situations. 

4.1.5  Analytical  Procedures 

The  procedures  used  to  develop  the  input  data  for  computer  simulation  are  a 
combination  of  manual  and  computer  assisted  (automated)  operations.  The 
product  of  these  operations  is  a  magnetic  tape  that  is  suitable  for  use  with 
the  analysis  "model." 

4.2  Execution  of  the  Computer  Simulation 

4.2.1  Object ive( s) 

The  objective(s)  to  be  met  in  performing  the  computer  simulation  (analysis) 
must  be  stated  clearly  and  concisely.  The  objective(s)  usually  will  pertain  to 
understanding  or  verifying  some  SLF  test  results  or  developing  an  initial 
understanding  of  some  very  complex  problem  that  may  subsequently  require  SLF 

testing. 

4.2.2  Cr iteria  (Appropriate  Regulation) 

The  criteria  to  be  met  in  performing  the  computer  simulation  (analysis)  must  be 
defined  along  with  the  basis  for  each  criterion. 

4.2.3  Data  Required 

The  computer  simulation  will  produce  output  data  from  the  "model"  that  describe 
equipment/system  performances  for  each  of  the  equipments/systems  that  are 
represented  in  the  scenario,  if  desired.  These  descriptions  of  performance 
will  be  statistical  estimates  of  equipment/system  performance,  according  to  the 
"instructions"  for  output  data  given  by  the  "model."  Typical  information  may 
include : 

a.  calculated  estimates  of  probabilities  of  correct  performance  for 
equipments/systems  of  interest  or  all  equipments/systems  in  the 

seenar  io 

b.  calculated  aggregations  of  the  estimates  of  probabilities  of 
correct  performance  for  equipments/systems  of  interest  in  the 

scenario 

c.  calculated  estimates  of  received  signal  level  (RSL)  for 
individual  equipments/systems  of  interest  in  the  scenario. 


. 4  Data  Acquisition  Procedure 


The  output  data  for  the  computer  simulation  will  be  presented  as  printed  and/or 
plotted  information  that  is  formatted  to  provide  the  required  information  about 

the  scenario  "snapshot." 


d .  1.5  Ana  ly t ical  Procedures 


'he  analytical  procedures  are  the  assembly  of  computer  subroutines  that 
const  i*  ate  the  "model"  which  is  an  implementation  of  the  algorithms  developed 


"equired  analyses  of  equipment/system  performances. 


F  FIELD  FACILITY  TESTS 


.  or m  tr. 


■  st  s  s bo 
.  Lmulat ion 

•  erf  ;,rmare 


est  'ana ly 
too  that 


•xoensive, 


on,  with  the  use  of  subsections,  will  define  and  describe  the  tests 
oe  conducted  to  verify  or  supplement  understanding  of  system 
c  observed  during  SLF  tests  and/or  from  computer  simulation.  Field 
ests  usually  will  be  the  tests  of  "last  resort."  That  is,  field 
aid  be  considered  only  (1)  when  other  test  modes  and/or  computer 
fail  to  answer  sufficiently  the  questions  being  asked  concerning 
e  of  the  system  being  tested  or  (2)  when  it  is  clear  that  these  other 
sis  modes  are  inadequate  to  perform  the  evaluation  of  system  perform- 
is  required.  Field  facility  tests  may  become  very  involved  and 
Therefore,  it  always  will  be  important  to  carefully  plan  any  field 
are  determined  to  be  necessary,  so  that  only  the  minimum  amount  of 
conducted  sufficient  to  respond  to  the  objectives  and  criteria  of 


a re  test  System,  Subsystem,  and/or  Component  Check-outs 


Lb  jeet Ives 


Ihe  objectives  are  to  determine  that  the  system,  subsystem,  and/or  components 
:.c  be  tested  as  well  as  the  systems,  subsystems,  and/or  components  that  will 


comprise  the  environment  for  the  test(s)  are  complete  and  in  normal  operating 
condition  and  that  any  and  all  required  support  equipments  are  available, 
complete ,  and  in  normal  operating  condition  prior  to  start  of  the  test(s). 


5.1.  2  Criteria  'Appropr iate  Regulation) 


The  system,  subsystem,  or  component  to  be  tested  shall  be 
complete  and  in  normal  operating  condition  prior  to  start  of  the 

test ( s)  . 


Any  support  equipment  required  for  the  system,  subsystem,  or 
'omponent  to  be  tested  shall  be  available,  complete,  and  in 
normal  operating  condition  prior  to  start  of  the  test(s). 


The  systems,  subsystems,  and/or  components  that  will  comprise  the 
environment  for  the  test(s)  shall  be  complete  and  in  normal 
operating  condition  prior  to  start  of  the  test(s). 
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d.  Any  and  all  support  equipment  required  for  the  systems, 
subsystems,  and/or  components  that  will  comprise  the  environment 
for  the  test(s)  shall  be  available,  complete,  and  in  normal 
operating  condition  prior  to  start  of  the  test(s). 

5.1.3  Data  Required 

a.  record  of  discrepancies  existent  for  the  system,  subsystem,  or 
component  to  be  tested 

b.  record  of  discrepancies  existent  for  required  support  equipment 
for  the  system,  subsystem,  or  component  to  be  tested 

c.  photographic  documentation  of  any  physical  damage  existent  for 
the  system,  subsystem,  or  component  to  be  tested  and/or  the 
support  equipment  required  for  the  system,  subsystem,  or 
component  to  be  tested 

d.  record  of  all  pretest  adjustments  and  repairs  performed  and  all 
performance  checks  not  met  for  the  system  to  be  tested  and  any 
required  support  equipment 

e.  record  of  discrepancies  existent  for  the  systems,  subsystems, 
and/or  components  that  will  comprise  the  environment  for  the 

test( s) 

f.  record  of  discrepancies  existent  for  required  support  equipment 
for  the  systems,  subsystems,  and/or  components  that  will  comprise 
the  environment  for  the  test(s). 

5.1.^  Data  Acquisition  Procedure 

The  test  officer  and  other  technical  and  maintenance  personnel,  as  required, 
will  conduct  the  pretest  system,  subsystem,  and/or  component  check-outs.  These 
check-outs  will  include: 

a.  Unpack  and  inventory  the  system,  subsystem,  or  component  to  be 
tested  and  all  required  support  equipment,  and  compare  the 
contents  with  packing  list  to  determine  if  any  discrepancies 
exist. 

b.  Inspect  the  system,  subsystem,  or  component  to  be  tested  and  all 
required  support  equipment  for  evidence  of  physical  damage. 

c.  Perform  a  pretest  operational  performance  verification  check  on 
t'ne  system,  subsystem,  or  component  to  be  tested  and  all  required 
support  equipment,  as  appropriate. 

d.  Adjust  and/or  repair  each  discrepant  condition,  if  possible. 

e.  Record  all  discrepant  conditions,  including  those  removed  by 
adjustment  and/or  repair  as  well  as  those  not  removed,  for  the 
system,  subsystem,  or  component  to  be  tested. 
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f.  Unpack  and  inventory  the  systems,  subsystems,  and/or  components 
that  will  comprise  the  environment  for  the  test(s)  and  all 
required  support  equipment,  and  compare  the  contents  with  packing 
lists  to  determine  if  any  discrepancies  exist. 

g.  Inspect  the  systems,  subsystems,  and/or  components  that  will 
comprise  the  environment  for  the  test(s)  and  all  required  support 
equipment  for  evidence  of  physical  damage. 

h.  Perform  a  pretest  operational  performance  verification  check  on 
each  system,  subsystem,  and/or  component  that  will  comprise  the 
environment  for  the  test(s)  and  all  required  support  equipment, 
as  appropriate. 

i.  Adjust  and/or  repair  each  discrepant  condition,  if  possible. 

j.  Record  all  discrepant  conditions  that  remain  uncorrected  for  the 
systems,  subsystems,  and/or  components  that  will  comprise  the 
environment  for  the  test(s). 

5 . ’ . 3  Ana ly t ical  Procedure 


The  data  will  be  used  to  assist  the  test  officer  in  determining  if  the  system, 
subsystem,  or  component  to  be  tested  and  all  associated,  required  support 
equipment,  ss  well  as  all  systems,  subsystems,  and/or  components  that  will 
comprise  the  environment  for  the  test  along  with  all  required  support 
quipment,  are  complete,  undamaged,  in  operating  condition,  and  ready  for 
est  i  rig. 


5 . 1  Execution  of  the  Field  Facility  Test( s) 

5.2.1  Objective( s) 

The  objectize(s)  of  field  facility  testing  must  be  defined  in  one  or  more 
short,  clear  statement( s) .  One  might  wish  to  measure  user-oriented 
performance,  however,  the  implementation  of  interface  monitors  in  field  testing 
will  be  challenging.  Or  one  might  wish  to  measure  only  some  of  the  many 
possible  engineering-oriented  parameters.  In  other  words,  different  objectives 
may  be  neceosary  for  different  emphases  in  the  field  facility  testing. 

5.2. 2  Cr iter ia  (Appropriate  Regulation) 

The  criteria  to  be  met  in  performing  the  field  facility  test(s)  must  be  defined 

along  with  the  basis  for  each  criterion. 

5.2.3  Data  Required 

Data  required  to  define  system  performance  (engineering-oriented 
spec  if icat ions)  normally  will  be  system  specific,  i.e.,  different  sets  of  data 
for  different  systems.  The  types  of  data  that  may  be  of  interest  for 
electronic  surveillance  systems  include  the  following: 


a.  system  accuracy  in  measuring  carrier  frequency  and  modulation 
characteristics,  e.g.,  pulse  width,  pulse  repetition  frequency, 
or  pulse  repetition  interval,  etc.,  in  a  benign  environment  as 
well  as  in  the  operational  or  stressed  environment  created  by  the 
many  other  systems,  subsystems,  and/or  components  deployed  for 
the  field  test(s) 

b.  bearing  accuracy  while  operating  in  a  benign  environment  a3  well 
as  in  the  operational  or  stressed  environment  created  by  the  many 
other  systems,  subsystems,  and/or  components  deployed  for  the 
field  test(s) 

c.  elliptical  error  accuracy  ( intersections  of  multiple  bearings 
from  multiple  locations)  while  operating  in  a  benign  environment 
as  well  as  in  the  operational  or  stressed  environment  created  by 
the  many  other  systems,  subsystems,  and/or  components  deployed 
for  the  field  test(s) 

d.  signal  processing  times  while  operating  in  a  benign  environment 
as  well  as  in  the  operational  or  stressed  environment  created  by 
the  many  other  systems,  subsystems,  and/or  components  denloyed 
for  the  field  test(s) 

e.  signal  sorting  capabilities  while  operating  in  a  benign 

environment  as  well  as  in  the  operational  or  stressed  environment 
created  by  the  many  other  systems,  subsystems,  and/or  components 
deployed  for  the  field  test(s) 

f.  speed  in  recording  and/or  printing  output  Information  while 
operating  in  a  benign  environment  as  well  as  in  the  operational 
or  stressed  environment  created  by  the  many  other  systems, 
subsystems,  and/or  components  deployed  for  the  field  test(s) 

g.  self-calibration  capabilities  while  operating  in  a  benign 

environment  as  well  as  in  the  operational  or  stressed  environment 
created  by  the  many  other  systems,  subsystems,  and/or  components 
deployed  for  the  field  test(s) 

h.  self-test  capabilities  while  operating  in  a  benign  environment  as 

well  as  in  the  operational  or  stressed  environment  created  by  the 
many  other  systems,  subsystems,  and/or  components  deployed  for 

the  field  test(s) 

i.  data  link  capabilities  while  operating  in  a  benign  environment  as 

well  as  in  the  operational  or  stressed  environment  created  by  the 
many  other  systems,  subsystems,  and/or  components  deployed  for 

the  field  test(s). 

5.2.4  Data  Acquisition  Procedure 

The  data  will  be  acquired  using  field  facility  instrumentation  that  is 
installed  and  operated  in  accordance  with  good  engineering  practices. 
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.Jo  Anal yt  i  cal  Procedures 

'he  data  will  be  reduced,  analyzed,  and  presented  in  formats  that  allow  for 
»sy  comparison  with  the  data  taken  during  other  modes  of  testing  (e.g.,  bench 
■ests,  SLF  tests,  and/or  computer  simulation).  These  formats  also  will  be 
consistent  with  appropriate  and  applicable  standards  and  good  engineering 

iract  i  ces . 
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